Feb 26 13:50:51 shakti kernel: [89048.423200] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:50:51 shakti kernel: [89048.423559]
Feb 26 13:50:51 shakti kernel: [89048.423660] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:50:51 shakti kernel: [89048.423791] EIP: 0060:[<f9114a02>] EFLAGS: 00000202 CPU: 1
Feb 26 13:50:51 shakti kernel: [89048.424347] EIP is at tcp_packet+0x9b2/0xbb0 [nf_conntrack]
Feb 26 13:50:51 shakti kernel: [89048.424484] EAX: 00000000 EBX: 0000fa80 ECX: ec070ce4 EDX: ec070ce4
Feb 26 13:50:51 shakti kernel: [89048.424608] ESI: 6a151a34 EDI: ec070c30 EBP: 00000001 ESP: df9cf680
Feb 26 13:50:51 shakti kernel: [89048.424724]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:50:51 shakti kernel: [89048.424849] CR0: 80050033 CR2: b7d521a8 CR3: 32491000 CR4: 000006b0
Feb 26 13:50:51 shakti kernel: [89048.424956] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:50:51 shakti kernel: [89048.425066] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:50:51 shakti kernel: [89048.425187]  [<f8974cad>] e100_disable_irq+0x3d/0x60 [e100]
Feb 26 13:50:51 shakti kernel: [89048.425580]  [<f8975925>] e100_intr+0xb5/0xc0 [e100]
Feb 26 13:50:51 shakti kernel: [89048.426126]  [<c0108dff>] common_interrupt+0x23/0x28
Feb 26 13:50:51 shakti kernel: [89048.426409]  [<f91103fc>] __nf_conntrack_find+0x1c/0x110 [nf_conntrack]
Feb 26 13:50:51 shakti kernel: [89048.427036]  [<f9114050>] tcp_packet+0x0/0xbb0 [nf_conntrack]
Feb 26 13:50:51 shakti kernel: [89048.427663]  [<f91117b1>] nf_conntrack_in+0x201/0x4d0 [nf_conntrack]
Feb 26 13:50:51 shakti kernel: [89048.428488]  [<f91021d0>] ipv4_conntrack_local+0x0/0x70 [nf_conntrack_ipv4]
Feb 26 13:50:51 shakti kernel: [89048.428815]  [<c02cfa97>] nf_iterate+0x57/0x80
Feb 26 13:50:51 shakti kernel: [89048.429003]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:50:51 shakti kernel: [89048.429204]  [<c02cfb98>] nf_hook_slow+0x98/0xd0
Feb 26 13:50:51 shakti kernel: [89048.429408]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:50:51 shakti kernel: [89048.429628]  [<c02dae47>] ip_queue_xmit+0x1e7/0x430
Feb 26 13:50:51 shakti kernel: [89048.429827]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:50:51 shakti kernel: [89048.430035]  [<c032ced0>] notifier_call_chain+0x30/0x60
Feb 26 13:50:51 shakti kernel: [89048.430273]  [<c032cf27>] atomic_notifier_call_chain+0x17/0x20
Feb 26 13:50:51 shakti kernel: [89048.430496]  [<c01916fe>] __slab_alloc+0x2ee/0x4a0
Feb 26 13:50:51 shakti kernel: [89048.430685]  [<c01916fe>] __slab_alloc+0x2ee/0x4a0
Feb 26 13:50:51 shakti kernel: [89048.430939]  [<c02ebc67>] tcp_transmit_skb+0x497/0x820
Feb 26 13:50:51 shakti kernel: [89048.431264]  [<c02ed6da>] __tcp_push_pending_frames+0x11a/0x8f0
Feb 26 13:50:51 shakti kernel: [89048.431639]  [<c02eb297>] tcp_current_mss+0x77/0x110
Feb 26 13:50:51 shakti kernel: [89048.431855]  [<c02ea984>] tcp_rcv_established+0x434/0x950
Feb 26 13:50:51 shakti kernel: [89048.432108]  [<c0315510>] __xfrm_policy_check+0x3e0/0x570
Feb 26 13:50:51 shakti kernel: [89048.432302]  [<c0314f10>] xfrm_policy_lookup+0x0/0x220
Feb 26 13:50:51 shakti kernel: [89048.432476]  [<f90f76cb>] ipt_do_table+0x20b/0x4b0 [ip_tables]
Feb 26 13:50:51 shakti kernel: [89048.432754]  [<c02f1590>] tcp_v4_do_rcv+0x340/0x6e0
Feb 26 13:50:51 shakti kernel: [89048.432957]  [<c02d30c9>] ip_route_input+0x39/0xdf0
Feb 26 13:50:51 shakti kernel: [89048.433148]  [<f9114050>] tcp_packet+0x0/0xbb0 [nf_conntrack]
Feb 26 13:50:51 shakti kernel: [89048.433835]  [<f911550d>] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack]
Feb 26 13:50:51 shakti kernel: [89048.434476]  [<f91021a9>] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4]
Feb 26 13:50:51 shakti kernel: [89048.434776]  [<f9102180>] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4]
Feb 26 13:50:51 shakti kernel: [89048.435082]  [<c02f417d>] tcp_v4_rcv+0x7bd/0x9a0
Feb 26 13:50:51 shakti kernel: [89048.435366]  [<c02d5f59>] ip_local_deliver_finish+0xf9/0x210
Feb 26 13:50:51 shakti kernel: [89048.435589]  [<c02d5bef>] ip_rcv_finish+0xff/0x370
Feb 26 13:50:51 shakti kernel: [89048.435795]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:50:51 shakti kernel: [89048.436018]  [<c02d632a>] ip_rcv+0x1ca/0x290
Feb 26 13:50:51 shakti kernel: [89048.436208]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:50:51 shakti kernel: [89048.436397]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:50:51 shakti kernel: [89048.436574]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:50:51 shakti kernel: [89048.436823]  [<f8b0bd17>] e1000_receive_skb+0x47/0x1a0 [e1000e]
Feb 26 13:50:51 shakti kernel: [89048.437374]  [<f8b0f264>] e1000_clean_rx_irq+0x124/0x300 [e1000e]
Feb 26 13:50:51 shakti kernel: [89048.437904]  [<f8b0f1b9>] e1000_clean_rx_irq+0x79/0x300 [e1000e]
Feb 26 13:50:51 shakti kernel: [89048.438537]  [<f8b0aca4>] e1000_clean+0x54/0x220 [e1000e]
Feb 26 13:50:51 shakti kernel: [89048.439041]  [<c02c5dfc>] __qdisc_run+0x7c/0x1a0
Feb 26 13:50:51 shakti kernel: [89048.439238]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:50:51 shakti kernel: [89048.439491]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:50:51 shakti kernel: [89048.439716]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:50:51 shakti kernel: [89048.439902]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:50:51 shakti kernel: [89048.440058]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:50:51 shakti kernel: [89048.440263]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:50:51 shakti kernel: [89048.440520]  [<f8899de6>] xor_sse_4+0x2b6/0x360 [xor]
Feb 26 13:50:51 shakti kernel: [89048.440919]  [<f8898b15>] xor_blocks+0x75/0xa0 [xor]
Feb 26 13:50:51 shakti kernel: [89048.441173]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.441521]  [<f88942fb>] async_xor+0x2fb/0x350 [async_xor]
Feb 26 13:50:51 shakti kernel: [89048.441761]  [<c0148c12>] enqueue_hrtimer+0x72/0x100
Feb 26 13:50:51 shakti kernel: [89048.442028]  [<f88e5d50>] ops_complete_write+0x0/0x60 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.442394]  [<f88e6181>] ops_run_postxor+0xd1/0x160 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.442755]  [<f88e5d50>] ops_complete_write+0x0/0x60 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.443202]  [<f88e7330>] handle_stripe5+0x1040/0x1590 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.443565]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:50:51 shakti kernel: [89048.443954]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.444433]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.444849]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:50:51 shakti kernel: [89048.445063]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.445578]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:50:51 shakti kernel: [89048.445797]  [<f88ab0c6>] md_wakeup_thread+0x26/0x30 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.446285]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.446624]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.447103]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:50:51 shakti kernel: [89048.447485]  [<c0108dff>] common_interrupt+0x23/0x28
Feb 26 13:50:51 shakti kernel: [89048.447697]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.448200]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.448681]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.449163]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:50:51 shakti kernel: [89048.449387]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:50:51 shakti kernel: [89048.449862]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:50:51 shakti kernel: [89048.450018]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:50:51 shakti kernel: [89048.450182]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:50:51 shakti kernel: [89048.450397]  =======================
Feb 26 13:51:02 shakti kernel: [89060.232012] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:51:02 shakti kernel: [89060.232340]
Feb 26 13:51:02 shakti kernel: [89060.232444] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:51:02 shakti kernel: [89060.232575] EIP: 0060:[<c0191948>] EFLAGS: 00000286 CPU: 1
Feb 26 13:51:02 shakti kernel: [89060.232683] EIP is at kfree+0x98/0xe0
Feb 26 13:51:02 shakti kernel: [89060.232798] EAX: 00000286 EBX: e21b6230 ECX: d5831800 EDX: d5831c00
Feb 26 13:51:02 shakti kernel: [89060.232927] ESI: 00000000 EDI: f8a6a374 EBP: c02adca8 ESP: df9cfbc4
Feb 26 13:51:02 shakti kernel: [89060.233053]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:51:02 shakti kernel: [89060.233204] CR0: 8005003b CR2: b7d521a8 CR3: 32491000 CR4: 000006b0
Feb 26 13:51:02 shakti kernel: [89060.233305] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:51:02 shakti kernel: [89060.233409] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:51:02 shakti kernel: [89060.233487]  [<c02adca8>] __kfree_skb+0x8/0x80
Feb 26 13:51:02 shakti kernel: [89060.233706]  [<c02adca8>] __kfree_skb+0x8/0x80
Feb 26 13:51:02 shakti kernel: [89060.233905]  [<f8b0a573>] e1000_put_txbuf+0x23/0x30 [e1000e]
Feb 26 13:51:02 shakti kernel: [89060.234447]  [<f8b0a6fe>] e1000_clean_tx_irq+0xde/0x330 [e1000e]
Feb 26 13:51:02 shakti kernel: [89060.235002]  [<f8975c0a>] e100_tx_clean+0x5a/0x100 [e100]
Feb 26 13:51:02 shakti kernel: [89060.235390]  [<c02adca8>] __kfree_skb+0x8/0x80
Feb 26 13:51:02 shakti kernel: [89060.235589]  [<f8975c0a>] e100_tx_clean+0x5a/0x100 [e100]
Feb 26 13:51:02 shakti kernel: [89060.235955]  [<c02af242>] __netdev_alloc_skb+0x22/0x50
Feb 26 13:51:02 shakti kernel: [89060.236215]  [<c014f3a8>] tick_program_event+0x38/0x60
Feb 26 13:51:02 shakti kernel: [89060.236427]  [<f8975d27>] e100_poll+0x77/0x2f0 [e100]
Feb 26 13:51:02 shakti kernel: [89060.236802]  [<f8b0ac89>] e1000_clean+0x39/0x220 [e1000e]
Feb 26 13:51:02 shakti kernel: [89060.237375]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:51:02 shakti kernel: [89060.237658]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:02 shakti kernel: [89060.237898]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:51:02 shakti kernel: [89060.238087]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:51:02 shakti kernel: [89060.238242]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:51:02 shakti kernel: [89060.238460]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:51:02 shakti kernel: [89060.238738]  [<c020ed80>] generic_make_request+0x220/0x4c0
Feb 26 13:51:02 shakti kernel: [89060.238966]  [<c010a940>] do_IRQ+0x40/0x70
Feb 26 13:51:02 shakti kernel: [89060.239334]  [<c014f3a8>] tick_program_event+0x38/0x60
Feb 26 13:51:02 shakti kernel: [89060.239575]  [<f88e6be0>] handle_stripe5+0x8f0/0x1590 [raid456]
Feb 26 13:51:02 shakti kernel: [89060.239945]  [<c0148c12>] enqueue_hrtimer+0x72/0x100
Feb 26 13:51:02 shakti kernel: [89060.240238]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:02 shakti kernel: [89060.240464]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.241090]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.241582]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:51:02 shakti kernel: [89060.241925]  [<c0148c12>] enqueue_hrtimer+0x72/0x100
Feb 26 13:51:02 shakti kernel: [89060.242195]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:02 shakti kernel: [89060.242418]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.242951]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:51:02 shakti kernel: [89060.243195]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:51:02 shakti kernel: [89060.243545]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.244036]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:51:02 shakti kernel: [89060.244424]  [<c0108dff>] common_interrupt+0x23/0x28
Feb 26 13:51:02 shakti kernel: [89060.244637]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.245147]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.245628]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.246123]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:51:02 shakti kernel: [89060.246354]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:02 shakti kernel: [89060.246840]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:51:02 shakti kernel: [89060.246995]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:51:02 shakti kernel: [89060.247164]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:51:02 shakti kernel: [89060.247384]  =======================
Feb 26 13:51:23 shakti kernel: [89080.742583] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:51:23 shakti kernel: [89080.742911]
Feb 26 13:51:23 shakti kernel: [89080.742995] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:51:23 shakti kernel: [89080.743113] EIP: 0060:[<f90f75bb>] EFLAGS: 00000246 CPU: 1
Feb 26 13:51:23 shakti kernel: [89080.743310] EIP is at ipt_do_table+0xfb/0x4b0 [ip_tables]
Feb 26 13:51:23 shakti kernel: [89080.743409] EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
Feb 26 13:51:23 shakti kernel: [89080.743540] ESI: f90f99d4 EDI: f922d538 EBP: f9228b60 ESP: df9cf89c
Feb 26 13:51:23 shakti kernel: [89080.743657]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:51:23 shakti kernel: [89080.743778] CR0: 8005003b CR2: b64d4000 CR3: 32491000 CR4: 000006b0
Feb 26 13:51:23 shakti kernel: [89080.743880] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:51:23 shakti kernel: [89080.743977] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:51:23 shakti kernel: [89080.744330]  [<f8cfe000>] ipt_local_out_hook+0x0/0x80 [iptable_filter]
Feb 26 13:51:23 shakti kernel: [89080.744551]  [<c02cfa97>] nf_iterate+0x57/0x80
Feb 26 13:51:23 shakti kernel: [89080.744761]  [<c02cfb98>] nf_hook_slow+0x98/0xd0
Feb 26 13:51:23 shakti kernel: [89080.744930]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:51:23 shakti kernel: [89080.745121]  [<c02dae47>] ip_queue_xmit+0x1e7/0x430
Feb 26 13:51:23 shakti kernel: [89080.745289]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:51:23 shakti kernel: [89080.745509]  [<c014cbca>] clocksource_get_next+0x3a/0x40
Feb 26 13:51:23 shakti kernel: [89080.745666]  [<c014b503>] update_wall_time+0x2e3/0x840
Feb 26 13:51:23 shakti kernel: [89080.745818]  [<c032cf27>] atomic_notifier_call_chain+0x17/0x20
Feb 26 13:51:23 shakti kernel: [89080.746013]  [<c01916fe>] __slab_alloc+0x2ee/0x4a0
Feb 26 13:51:23 shakti kernel: [89080.746159]  [<c01916fe>] __slab_alloc+0x2ee/0x4a0
Feb 26 13:51:23 shakti kernel: [89080.746306]  [<c012a88e>] try_to_wake_up+0x4e/0x350
Feb 26 13:51:23 shakti kernel: [89080.746553]  [<c02ebc67>] tcp_transmit_skb+0x497/0x820
Feb 26 13:51:23 shakti kernel: [89080.746890]  [<c02ed6da>] __tcp_push_pending_frames+0x11a/0x8f0
Feb 26 13:51:23 shakti kernel: [89080.747286]  [<c02eb297>] tcp_current_mss+0x77/0x110
Feb 26 13:51:23 shakti kernel: [89080.747471]  [<c02ea984>] tcp_rcv_established+0x434/0x950
Feb 26 13:51:23 shakti kernel: [89080.747706]  [<c0315510>] __xfrm_policy_check+0x3e0/0x570
Feb 26 13:51:23 shakti kernel: [89080.747864]  [<c0314f10>] xfrm_policy_lookup+0x0/0x220
Feb 26 13:51:23 shakti kernel: [89080.748019]  [<f90f76cb>] ipt_do_table+0x20b/0x4b0 [ip_tables]
Feb 26 13:51:23 shakti kernel: [89080.748297]  [<c02f1590>] tcp_v4_do_rcv+0x340/0x6e0
Feb 26 13:51:23 shakti kernel: [89080.748466]  [<c02d30c9>] ip_route_input+0x39/0xdf0
Feb 26 13:51:23 shakti kernel: [89080.748619]  [<f9114050>] tcp_packet+0x0/0xbb0 [nf_conntrack]
Feb 26 13:51:23 shakti kernel: [89080.749344]  [<f911550d>] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack]
Feb 26 13:51:23 shakti kernel: [89080.749967]  [<f91021a9>] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4]
Feb 26 13:51:23 shakti kernel: [89080.750235]  [<f9102180>] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4]
Feb 26 13:51:23 shakti kernel: [89080.750540]  [<c02f417d>] tcp_v4_rcv+0x7bd/0x9a0
Feb 26 13:51:23 shakti kernel: [89080.750820]  [<c02d5f59>] ip_local_deliver_finish+0xf9/0x210
Feb 26 13:51:23 shakti kernel: [89080.751017]  [<c02d5bef>] ip_rcv_finish+0xff/0x370
Feb 26 13:51:23 shakti kernel: [89080.751191]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:23 shakti kernel: [89080.751388]  [<c02d632a>] ip_rcv+0x1ca/0x290
Feb 26 13:51:23 shakti kernel: [89080.751547]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:23 shakti kernel: [89080.751702]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:51:23 shakti kernel: [89080.751865]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:51:23 shakti kernel: [89080.752098]  [<f8b0bd17>] e1000_receive_skb+0x47/0x1a0 [e1000e]
Feb 26 13:51:23 shakti kernel: [89080.752637]  [<f8b0f264>] e1000_clean_rx_irq+0x124/0x300 [e1000e]
Feb 26 13:51:23 shakti kernel: [89080.753147]  [<f8b0f1b9>] e1000_clean_rx_irq+0x79/0x300 [e1000e]
Feb 26 13:51:23 shakti kernel: [89080.753803]  [<f8b0aca4>] e1000_clean+0x54/0x220 [e1000e]
Feb 26 13:51:23 shakti kernel: [89080.754331]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:51:23 shakti kernel: [89080.754575]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:23 shakti kernel: [89080.754779]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:51:23 shakti kernel: [89080.754928]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:51:23 shakti kernel: [89080.755065]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:51:23 shakti kernel: [89080.755263]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:51:23 shakti kernel: [89080.755507]  [<c020eb9c>] generic_make_request+0x3c/0x4c0
Feb 26 13:51:23 shakti kernel: [89080.755714]  [<f88e6be0>] handle_stripe5+0x8f0/0x1590 [raid456]
Feb 26 13:51:23 shakti kernel: [89080.756057]  [<c016a6bc>] handle_fasteoi_irq+0x9c/0xe0
Feb 26 13:51:23 shakti kernel: [89080.756346]  [<c014be14>] getnstimeofday+0x34/0xf0
Feb 26 13:51:23 shakti kernel: [89080.756493]  [<c011c27c>] lapic_next_event+0xc/0x10
Feb 26 13:51:23 shakti kernel: [89080.756639]  [<c014e283>] clockevents_program_event+0xa3/0x150
Feb 26 13:51:23 shakti kernel: [89080.756982]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.757443]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:51:23 shakti kernel: [89080.757859]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:23 shakti kernel: [89080.758046]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.758543]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:51:23 shakti kernel: [89080.758751]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:51:23 shakti kernel: [89080.759059]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.759508]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:51:23 shakti kernel: [89080.759916]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.760417]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.760874]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.761337]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:51:23 shakti kernel: [89080.761535]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:23 shakti kernel: [89080.761987]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:51:23 shakti kernel: [89080.762122]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:51:23 shakti kernel: [89080.762275]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:51:23 shakti kernel: [89080.762468]  =======================
Feb 26 13:51:35 shakti kernel: [89092.551400] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:51:35 shakti kernel: [89092.551720]
Feb 26 13:51:35 shakti kernel: [89092.551824] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:51:35 shakti kernel: [89092.551956] EIP: 0060:[<c02adf72>] EFLAGS: 00000292 CPU: 1
Feb 26 13:51:35 shakti kernel: [89092.552082] EIP is at pskb_expand_head+0x152/0x160
Feb 26 13:51:35 shakti kernel: [89092.552203] EAX: 00000000 EBX: 00000022 ECX: 00010002 EDX: f4c07e80
Feb 26 13:51:35 shakti kernel: [89092.552354] ESI: 00000b50 EDI: e201dd80 EBP: e11ab230 ESP: df9cf7b4
Feb 26 13:51:35 shakti kernel: [89092.552469]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:51:35 shakti kernel: [89092.552591] CR0: 8005003b CR2: b64d4000 CR3: 00494000 CR4: 000006b0
Feb 26 13:51:35 shakti kernel: [89092.552696] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:51:35 shakti kernel: [89092.552791] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:51:35 shakti kernel: [89092.552865]  [<c02adfc5>] __pskb_pull_tail+0x45/0x2c0
Feb 26 13:51:35 shakti kernel: [89092.553148]  [<f8b0cb71>] e1000_xmit_frame+0x231/0xab0 [e1000e]
Feb 26 13:51:35 shakti kernel: [89092.553888]  [<c021e5ff>] strlcpy+0x1f/0x60
Feb 26 13:51:35 shakti kernel: [89092.554107]  [<f8b6343a>] packet_rcv_spkt+0x2a/0x1a0 [af_packet]
Feb 26 13:51:35 shakti kernel: [89092.554439]  [<f8b63410>] packet_rcv_spkt+0x0/0x1a0 [af_packet]
Feb 26 13:51:35 shakti kernel: [89092.554753]  [<c02b3fcc>] dev_hard_start_xmit+0x21c/0x2a0
Feb 26 13:51:35 shakti kernel: [89092.554971]  [<f911550d>] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack]
Feb 26 13:51:35 shakti kernel: [89092.555652]  [<c02c5ddb>] __qdisc_run+0x5b/0x1a0
Feb 26 13:51:35 shakti kernel: [89092.555846]  [<c02d9eb0>] ip_finish_output+0x0/0x2c0
Feb 26 13:51:35 shakti kernel: [89092.556076]  [<c02b65a7>] dev_queue_xmit+0x227/0x300
Feb 26 13:51:35 shakti kernel: [89092.556273]  [<c02d9eb0>] ip_finish_output+0x0/0x2c0
Feb 26 13:51:35 shakti kernel: [89092.556473]  [<c02d9ffc>] ip_finish_output+0x14c/0x2c0
Feb 26 13:51:35 shakti kernel: [89092.556722]  [<c02daedf>] ip_queue_xmit+0x27f/0x430
Feb 26 13:51:35 shakti kernel: [89092.556922]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:51:35 shakti kernel: [89092.557130]  [<c032ced0>] notifier_call_chain+0x30/0x60
Feb 26 13:51:35 shakti kernel: [89092.557390]  [<c01916fe>] __slab_alloc+0x2ee/0x4a0
Feb 26 13:51:35 shakti kernel: [89092.557576]  [<c01916fe>] __slab_alloc+0x2ee/0x4a0
Feb 26 13:51:35 shakti kernel: [89092.557770]  [<c018fec3>] add_partial_tail+0x13/0x40
Feb 26 13:51:35 shakti kernel: [89092.558005]  [<c02ebc67>] tcp_transmit_skb+0x497/0x820
Feb 26 13:51:35 shakti kernel: [89092.558327]  [<c02ed6da>] __tcp_push_pending_frames+0x11a/0x8f0
Feb 26 13:51:35 shakti kernel: [89092.558693]  [<c02eb297>] tcp_current_mss+0x77/0x110
Feb 26 13:51:35 shakti kernel: [89092.558906]  [<c02ea984>] tcp_rcv_established+0x434/0x950
Feb 26 13:51:35 shakti kernel: [89092.559161]  [<c0315510>] __xfrm_policy_check+0x3e0/0x570
Feb 26 13:51:35 shakti kernel: [89092.559357]  [<c0314f10>] xfrm_policy_lookup+0x0/0x220
Feb 26 13:51:35 shakti kernel: [89092.559533]  [<f90f76cb>] ipt_do_table+0x20b/0x4b0 [ip_tables]
Feb 26 13:51:35 shakti kernel: [89092.559812]  [<c02f1590>] tcp_v4_do_rcv+0x340/0x6e0
Feb 26 13:51:35 shakti kernel: [89092.560016]  [<c02d30c9>] ip_route_input+0x39/0xdf0
Feb 26 13:51:35 shakti kernel: [89092.560209]  [<f9114050>] tcp_packet+0x0/0xbb0 [nf_conntrack]
Feb 26 13:51:35 shakti kernel: [89092.560900]  [<f911550d>] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack]
Feb 26 13:51:35 shakti kernel: [89092.561549]  [<f91021a9>] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4]
Feb 26 13:51:35 shakti kernel: [89092.561847]  [<f9102180>] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4]
Feb 26 13:51:35 shakti kernel: [89092.562147]  [<c02f417d>] tcp_v4_rcv+0x7bd/0x9a0
Feb 26 13:51:35 shakti kernel: [89092.562415]  [<c02d5f59>] ip_local_deliver_finish+0xf9/0x210
Feb 26 13:51:35 shakti kernel: [89092.562634]  [<c02d5bef>] ip_rcv_finish+0xff/0x370
Feb 26 13:51:35 shakti kernel: [89092.562836]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:35 shakti kernel: [89092.563053]  [<c02d632a>] ip_rcv+0x1ca/0x290
Feb 26 13:51:35 shakti kernel: [89092.563245]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:35 shakti kernel: [89092.563436]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:51:35 shakti kernel: [89092.563612]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:51:35 shakti kernel: [89092.563861]  [<f8b0bd17>] e1000_receive_skb+0x47/0x1a0 [e1000e]
Feb 26 13:51:35 shakti kernel: [89092.564418]  [<f8b0f264>] e1000_clean_rx_irq+0x124/0x300 [e1000e]
Feb 26 13:51:35 shakti kernel: [89092.564958]  [<f8b0f1b9>] e1000_clean_rx_irq+0x79/0x300 [e1000e]
Feb 26 13:51:35 shakti kernel: [89092.565597]  [<f8b0aca4>] e1000_clean+0x54/0x220 [e1000e]
Feb 26 13:51:35 shakti kernel: [89092.566150]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:51:35 shakti kernel: [89092.566404]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:35 shakti kernel: [89092.566631]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:51:35 shakti kernel: [89092.566817]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:51:35 shakti kernel: [89092.566975]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:51:35 shakti kernel: [89092.567158]  [<c0127857>] enqueue_task_fair+0x27/0x30
Feb 26 13:51:35 shakti kernel: [89092.567352]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:51:35 shakti kernel: [89092.567594]  [<f883c035>] async_tx_find_channel+0x35/0x40 [async_tx]
Feb 26 13:51:35 shakti kernel: [89092.567881]  [<f8844020>] async_memcpy+0x20/0x1d0 [async_memcpy]
Feb 26 13:51:35 shakti kernel: [89092.568129]  [<f88e6181>] ops_run_postxor+0xd1/0x160 [raid456]
Feb 26 13:51:35 shakti kernel: [89092.568501]  [<f88e4e99>] async_copy_data+0x79/0x140 [raid456]
Feb 26 13:51:35 shakti kernel: [89092.568920]  [<f88e69ec>] handle_stripe5+0x6fc/0x1590 [raid456]
Feb 26 13:51:35 shakti kernel: [89092.569521]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.570010]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:51:35 shakti kernel: [89092.570438]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:35 shakti kernel: [89092.570656]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.571185]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:51:35 shakti kernel: [89092.571409]  [<f88ab0c6>] md_wakeup_thread+0x26/0x30 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.571908]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:51:35 shakti kernel: [89092.572255]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.572748]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:51:35 shakti kernel: [89092.573159]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.573679]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.574166]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.574660]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:51:35 shakti kernel: [89092.574884]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:35 shakti kernel: [89092.575368]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:51:35 shakti kernel: [89092.575525]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:51:35 shakti kernel: [89092.575691]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:51:35 shakti kernel: [89092.575905]  =======================
Feb 26 13:51:47 shakti kernel: [89104.360210] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:51:47 shakti kernel: [89104.360537]
Feb 26 13:51:47 shakti kernel: [89104.360639] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:51:47 shakti kernel: [89104.360767] EIP: 0060:[<f90f75ea>] EFLAGS: 00000287 CPU: 1
Feb 26 13:51:47 shakti kernel: [89104.360982] EIP is at ipt_do_table+0x12a/0x4b0 [ip_tables]
Feb 26 13:51:47 shakti kernel: [89104.361099] EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 0000000c
Feb 26 13:51:47 shakti kernel: [89104.361237] ESI: f90f99d4 EDI: f91e6094 EBP: f91e7300 ESP: df9cfa5c
Feb 26 13:51:47 shakti kernel: [89104.361359]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:51:47 shakti kernel: [89104.361499] CR0: 8005003b CR2: b64d4000 CR3: 00494000 CR4: 000006b0
Feb 26 13:51:47 shakti kernel: [89104.361599] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:51:47 shakti kernel: [89104.361693] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:51:47 shakti kernel: [89104.362050]  [<f90e6000>] ipt_route_hook+0x0/0x20 [iptable_mangle]
Feb 26 13:51:47 shakti kernel: [89104.362306]  [<c02cfa97>] nf_iterate+0x57/0x80
Feb 26 13:51:47 shakti kernel: [89104.362557]  [<c02cfb98>] nf_hook_slow+0x98/0xd0
Feb 26 13:51:47 shakti kernel: [89104.362764]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:47 shakti kernel: [89104.363000]  [<c02d632a>] ip_rcv+0x1ca/0x290
Feb 26 13:51:47 shakti kernel: [89104.363193]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:47 shakti kernel: [89104.363385]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:51:47 shakti kernel: [89104.363587]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:51:47 shakti kernel: [89104.363855]  [<f8b0bd17>] e1000_receive_skb+0x47/0x1a0 [e1000e]
Feb 26 13:51:47 shakti kernel: [89104.364420]  [<f8b0f264>] e1000_clean_rx_irq+0x124/0x300 [e1000e]
Feb 26 13:51:47 shakti kernel: [89104.364953]  [<f8b0f1b9>] e1000_clean_rx_irq+0x79/0x300 [e1000e]
Feb 26 13:51:47 shakti kernel: [89104.365608]  [<f8b0aca4>] e1000_clean+0x54/0x220 [e1000e]
Feb 26 13:51:47 shakti kernel: [89104.366106]  [<c02c5dfc>] __qdisc_run+0x7c/0x1a0
Feb 26 13:51:47 shakti kernel: [89104.366307]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:51:47 shakti kernel: [89104.366569]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:47 shakti kernel: [89104.366800]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:51:47 shakti kernel: [89104.366983]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:51:47 shakti kernel: [89104.367139]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:51:47 shakti kernel: [89104.367330]  [<c0216eb3>] deadline_add_request+0x23/0x60
Feb 26 13:51:47 shakti kernel: [89104.367520]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:51:47 shakti kernel: [89104.367765]  [<c021007b>] blk_queue_invalidate_tags+0x2b/0x40
Feb 26 13:51:47 shakti kernel: [89104.367965]  [<c020007b>] security_netlbl_secattr_to_sid+0x23b/0x250
Feb 26 13:51:47 shakti kernel: [89104.368174]  [<c0212048>] __make_request+0xd8/0x5b0
Feb 26 13:51:47 shakti kernel: [89104.368414]  [<c014f3a8>] tick_program_event+0x38/0x60
Feb 26 13:51:47 shakti kernel: [89104.368668]  [<c0148c12>] enqueue_hrtimer+0x72/0x100
Feb 26 13:51:47 shakti kernel: [89104.368894]  [<c020ed85>] generic_make_request+0x225/0x4c0
Feb 26 13:51:47 shakti kernel: [89104.369088]  [<c01498a0>] run_hrtimer_softirq+0xe0/0xf0
Feb 26 13:51:47 shakti kernel: [89104.369310]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:47 shakti kernel: [89104.369589]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:51:47 shakti kernel: [89104.369870]  [<f88e6be0>] handle_stripe5+0x8f0/0x1590 [raid456]
Feb 26 13:51:47 shakti kernel: [89104.370222]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:51:47 shakti kernel: [89104.370507]  [<c014be14>] getnstimeofday+0x34/0xf0
Feb 26 13:51:47 shakti kernel: [89104.370689]  [<c011c27c>] lapic_next_event+0xc/0x10
Feb 26 13:51:47 shakti kernel: [89104.370855]  [<c014e283>] clockevents_program_event+0xa3/0x150
Feb 26 13:51:47 shakti kernel: [89104.371179]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.371664]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:51:47 shakti kernel: [89104.372000]  [<c0148c12>] enqueue_hrtimer+0x72/0x100
Feb 26 13:51:47 shakti kernel: [89104.372267]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:47 shakti kernel: [89104.372489]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.373015]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:51:47 shakti kernel: [89104.373255]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:51:47 shakti kernel: [89104.373600]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.374085]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:51:47 shakti kernel: [89104.374508]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.375025]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.375507]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.375994]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:51:47 shakti kernel: [89104.376225]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:47 shakti kernel: [89104.376703]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:51:47 shakti kernel: [89104.376857]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:51:47 shakti kernel: [89104.377027]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:51:47 shakti kernel: [89104.377249]  =======================
Feb 26 13:51:58 shakti kernel: [89116.169014] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:51:58 shakti kernel: [89116.169336]
Feb 26 13:51:58 shakti kernel: [89116.169438] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:51:58 shakti kernel: [89116.169568] EIP: 0060:[<c02ed6b0>] EFLAGS: 00000297 CPU: 1
Feb 26 13:51:58 shakti kernel: [89116.169698] EIP is at __tcp_push_pending_frames+0xf0/0x8f0
Feb 26 13:51:58 shakti kernel: [89116.169823] EAX: d545c000 EBX: e2133600 ECX: 00000b50 EDX: e2133600
Feb 26 13:51:58 shakti kernel: [89116.169972] ESI: d545c000 EDI: 00000019 EBP: e2133620 ESP: df9cfa48
Feb 26 13:51:58 shakti kernel: [89116.170100]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:51:58 shakti kernel: [89116.170223] CR0: 8005003b CR2: b64d4000 CR3: 00494000 CR4: 000006b0
Feb 26 13:51:58 shakti kernel: [89116.170323] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:51:58 shakti kernel: [89116.170418] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:51:58 shakti kernel: [89116.170763]  [<c02eb297>] tcp_current_mss+0x77/0x110
Feb 26 13:51:58 shakti kernel: [89116.170988]  [<c02ea984>] tcp_rcv_established+0x434/0x950
Feb 26 13:51:58 shakti kernel: [89116.171256]  [<c0315510>] __xfrm_policy_check+0x3e0/0x570
Feb 26 13:51:58 shakti kernel: [89116.171449]  [<c0314f10>] xfrm_policy_lookup+0x0/0x220
Feb 26 13:51:58 shakti kernel: [89116.171626]  [<f90f76cb>] ipt_do_table+0x20b/0x4b0 [ip_tables]
Feb 26 13:51:58 shakti kernel: [89116.171922]  [<c02f1590>] tcp_v4_do_rcv+0x340/0x6e0
Feb 26 13:51:58 shakti kernel: [89116.172126]  [<c02d30c9>] ip_route_input+0x39/0xdf0
Feb 26 13:51:58 shakti kernel: [89116.172318]  [<f9114050>] tcp_packet+0x0/0xbb0 [nf_conntrack]
Feb 26 13:51:58 shakti kernel: [89116.173043]  [<f911550d>] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack]
Feb 26 13:51:58 shakti kernel: [89116.173702]  [<f91021a9>] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4]
Feb 26 13:51:58 shakti kernel: [89116.174008]  [<f9102180>] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4]
Feb 26 13:51:58 shakti kernel: [89116.174327]  [<c02f417d>] tcp_v4_rcv+0x7bd/0x9a0
Feb 26 13:51:58 shakti kernel: [89116.174625]  [<c02d5f59>] ip_local_deliver_finish+0xf9/0x210
Feb 26 13:51:58 shakti kernel: [89116.174855]  [<c02d5bef>] ip_rcv_finish+0xff/0x370
Feb 26 13:51:58 shakti kernel: [89116.175063]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:58 shakti kernel: [89116.175293]  [<c02d632a>] ip_rcv+0x1ca/0x290
Feb 26 13:51:58 shakti kernel: [89116.175485]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:51:58 shakti kernel: [89116.175677]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:51:58 shakti kernel: [89116.175857]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:51:58 shakti kernel: [89116.176117]  [<f8b0bd17>] e1000_receive_skb+0x47/0x1a0 [e1000e]
Feb 26 13:51:58 shakti kernel: [89116.176685]  [<f8b0f264>] e1000_clean_rx_irq+0x124/0x300 [e1000e]
Feb 26 13:51:58 shakti kernel: [89116.177230]  [<f8b0f1b9>] e1000_clean_rx_irq+0x79/0x300 [e1000e]
Feb 26 13:51:58 shakti kernel: [89116.177892]  [<f8b0aca4>] e1000_clean+0x54/0x220 [e1000e]
Feb 26 13:51:58 shakti kernel: [89116.178449]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:51:58 shakti kernel: [89116.178715]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:51:58 shakti kernel: [89116.178944]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:51:58 shakti kernel: [89116.179129]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:51:58 shakti kernel: [89116.179286]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:51:58 shakti kernel: [89116.179499]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:51:58 shakti kernel: [89116.179750]  [<c01b007b>] single_open+0xb/0x90
Feb 26 13:51:58 shakti kernel: [89116.179935]  [<f88800d8>] max_io_len+0x48/0xb0 [dm_mod]
Feb 26 13:51:58 shakti kernel: [89116.180416]  [<c0170cd5>] mempool_free+0x15/0xa0
Feb 26 13:51:58 shakti kernel: [89116.180629]  [<f88e302b>] return_io+0x2b/0x40 [raid456]
Feb 26 13:51:58 shakti kernel: [89116.180961]  [<f88e6c52>] handle_stripe5+0x962/0x1590 [raid456]
Feb 26 13:51:58 shakti kernel: [89116.181338]  [<c016a77b>] handle_edge_irq+0x7b/0x120
Feb 26 13:51:58 shakti kernel: [89116.181640]  [<c014be14>] getnstimeofday+0x34/0xf0
Feb 26 13:51:58 shakti kernel: [89116.181824]  [<c011c27c>] lapic_next_event+0xc/0x10
Feb 26 13:51:58 shakti kernel: [89116.181990]  [<c014e283>] clockevents_program_event+0xa3/0x150
Feb 26 13:51:58 shakti kernel: [89116.182322]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:58 shakti kernel: [89116.182816]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:51:58 shakti kernel: [89116.183156]  [<c012a88e>] try_to_wake_up+0x4e/0x350
Feb 26 13:51:58 shakti kernel: [89116.183440]  [<c0126a7b>] __wake_up_common+0x4b/0x80
Feb 26 13:51:58 shakti kernel: [89116.183667]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:51:58 shakti kernel: [89116.183894]  [<f88e3ace>] __release_stripe+0x10e/0x150 [raid456]
Feb 26 13:51:58 shakti kernel: [89116.184253]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:51:58 shakti kernel: [89116.184595]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:58 shakti kernel: [89116.185067]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:51:58 shakti kernel: [89116.185496]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:58 shakti kernel: [89116.186025]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:58 shakti kernel: [89116.186513]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:51:58 shakti kernel: [89116.187008]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:51:58 shakti kernel: [89116.187241]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:51:58 shakti kernel: [89116.187728]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:51:58 shakti kernel: [89116.187904]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:51:58 shakti kernel: [89116.188073]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:51:58 shakti kernel: [89116.188300]  =======================
Feb 26 13:52:10 shakti kernel: [89127.967830] BUG: soft lockup - CPU#1 stuck for 11s! [md0_raid5:2768]
Feb 26 13:52:10 shakti kernel: [89127.968150]
Feb 26 13:52:10 shakti kernel: [89127.968252] Pid: 2768, comm: md0_raid5 Not tainted (2.6.24-8-server #1)
Feb 26 13:52:10 shakti kernel: [89127.968387] EIP: 0060:[<f90f7670>] EFLAGS: 00000202 CPU: 1
Feb 26 13:52:10 shakti kernel: [89127.968602] EIP is at ipt_do_table+0x1b0/0x4b0 [ip_tables]
Feb 26 13:52:10 shakti kernel: [89127.968736] EAX: df9cf9e7 EBX: 0000009c ECX: 00000000 EDX: e251de40
Feb 26 13:52:10 shakti kernel: [89127.968866] ESI: f91e6edc EDI: f91e6e6c EBP: 00000070 ESP: df9cf970
Feb 26 13:52:10 shakti kernel: [89127.968984]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Feb 26 13:52:10 shakti kernel: [89127.969125] CR0: 8005003b CR2: b64d4000 CR3: 00494000 CR4: 000006b0
Feb 26 13:52:10 shakti kernel: [89127.969227] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Feb 26 13:52:10 shakti kernel: [89127.969330] DR6: ffff0ff0 DR7: 00000400
Feb 26 13:52:10 shakti kernel: [89127.969476]  [<f90f76cb>] ipt_do_table+0x20b/0x4b0 [ip_tables]
Feb 26 13:52:10 shakti kernel: [89127.969943]  [<f90e6000>] ipt_route_hook+0x0/0x20 [iptable_mangle]
Feb 26 13:52:10 shakti kernel: [89127.970198]  [<c02cfa97>] nf_iterate+0x57/0x80
Feb 26 13:52:10 shakti kernel: [89127.970448]  [<c02cfb98>] nf_hook_slow+0x98/0xd0
Feb 26 13:52:10 shakti kernel: [89127.970656]  [<c02d9eb0>] ip_finish_output+0x0/0x2c0
Feb 26 13:52:10 shakti kernel: [89127.970897]  [<c02db31c>] ip_output+0x8c/0xa0
Feb 26 13:52:10 shakti kernel: [89127.971091]  [<c02d9eb0>] ip_finish_output+0x0/0x2c0
Feb 26 13:52:10 shakti kernel: [89127.971299]  [<c02d8a2e>] ip_push_pending_frames+0x31e/0x410
Feb 26 13:52:10 shakti kernel: [89127.971519]  [<c02d85b0>] dst_output+0x0/0x10
Feb 26 13:52:10 shakti kernel: [89127.971758]  [<c02fc839>] icmp_send+0x289/0x410
Feb 26 13:52:10 shakti kernel: [89127.972039]  [<c0315510>] __xfrm_policy_check+0x3e0/0x570
Feb 26 13:52:10 shakti kernel: [89127.972231]  [<c0314f10>] xfrm_policy_lookup+0x0/0x220
Feb 26 13:52:10 shakti kernel: [89127.972503]  [<f917502d>] ctnetlink_conntrack_event+0x4d/0x930 [nf_conntrack_netlink]
Feb 26 13:52:10 shakti kernel: [89127.972905]  [<c02f7833>] __udp4_lib_lookup+0x23/0x120
Feb 26 13:52:10 shakti kernel: [89127.973140]  [<c02f88c7>] __udp4_lib_rcv+0x4e7/0x840
Feb 26 13:52:10 shakti kernel: [89127.973355]  [<f911550d>] nf_ct_deliver_cached_events+0x8d/0xc0 [nf_conntrack]
Feb 26 13:52:10 shakti kernel: [89127.973995]  [<f91021a9>] ipv4_confirm+0x29/0x50 [nf_conntrack_ipv4]
Feb 26 13:52:10 shakti kernel: [89127.974297]  [<f9102180>] ipv4_confirm+0x0/0x50 [nf_conntrack_ipv4]
Feb 26 13:52:10 shakti kernel: [89127.974582]  [<c02cfa97>] nf_iterate+0x57/0x80
Feb 26 13:52:10 shakti kernel: [89127.974749]  [<c02d5e60>] ip_local_deliver_finish+0x0/0x210
Feb 26 13:52:10 shakti kernel: [89127.974971]  [<c02cfb98>] nf_hook_slow+0x98/0xd0
Feb 26 13:52:10 shakti kernel: [89127.975177]  [<c02d5e60>] ip_local_deliver_finish+0x0/0x210
Feb 26 13:52:10 shakti kernel: [89127.975407]  [<c02d5f59>] ip_local_deliver_finish+0xf9/0x210
Feb 26 13:52:10 shakti kernel: [89127.975635]  [<c02d5bef>] ip_rcv_finish+0xff/0x370
Feb 26 13:52:10 shakti kernel: [89127.975844]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:52:10 shakti kernel: [89127.976066]  [<c02d632a>] ip_rcv+0x1ca/0x290
Feb 26 13:52:10 shakti kernel: [89127.976257]  [<c02d5af0>] ip_rcv_finish+0x0/0x370
Feb 26 13:52:10 shakti kernel: [89127.976447]  [<c02d6160>] ip_rcv+0x0/0x290
Feb 26 13:52:10 shakti kernel: [89127.976622]  [<c02b3839>] netif_receive_skb+0x379/0x440
Feb 26 13:52:10 shakti kernel: [89127.976874]  [<f8975dc9>] e100_poll+0x119/0x2f0 [e100]
Feb 26 13:52:10 shakti kernel: [89127.977228]  [<c014be14>] getnstimeofday+0x34/0xf0
Feb 26 13:52:10 shakti kernel: [89127.977393]  [<c011c27c>] lapic_next_event+0xc/0x10
Feb 26 13:52:10 shakti kernel: [89127.977562]  [<c014e283>] clockevents_program_event+0xa3/0x150
Feb 26 13:52:10 shakti kernel: [89127.977751]  [<f8975d27>] e100_poll+0x77/0x2f0 [e100]
Feb 26 13:52:10 shakti kernel: [89127.978141]  [<c02b5c2d>] net_rx_action+0x12d/0x210
Feb 26 13:52:10 shakti kernel: [89127.978327]  [<c014f3a8>] tick_program_event+0x38/0x60
Feb 26 13:52:10 shakti kernel: [89127.978564]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:52:10 shakti kernel: [89127.978789]  [<c0136b15>] do_softirq+0x55/0x60
Feb 26 13:52:10 shakti kernel: [89127.978974]  [<c0136dfd>] irq_exit+0x6d/0x80
Feb 26 13:52:10 shakti kernel: [89127.979129]  [<c011c825>] smp_apic_timer_interrupt+0x55/0x80
Feb 26 13:52:10 shakti kernel: [89127.979336]  [<c0108ebc>] apic_timer_interrupt+0x28/0x30
Feb 26 13:52:10 shakti kernel: [89127.979571]  [<c017007b>] generic_file_aio_write_nolock+0x7b/0xb0
Feb 26 13:52:10 shakti kernel: [89127.979775]  [<c017007b>] generic_file_aio_write_nolock+0x7b/0xb0
Feb 26 13:52:10 shakti kernel: [89127.979980]  [<f88804f2>] dec_pending+0xb2/0x1e0 [dm_mod]
Feb 26 13:52:10 shakti kernel: [89127.980482]  [<c0170d48>] mempool_free+0x88/0xa0
Feb 26 13:52:10 shakti kernel: [89127.980675]  [<c0170d48>] mempool_free+0x88/0xa0
Feb 26 13:52:10 shakti kernel: [89127.980907]  [<f88807e2>] clone_endio+0x62/0xb0 [dm_mod]
Feb 26 13:52:10 shakti kernel: [89127.981438]  [<f88e302b>] return_io+0x2b/0x40 [raid456]
Feb 26 13:52:10 shakti kernel: [89127.981764]  [<f88e6c52>] handle_stripe5+0x962/0x1590 [raid456]
Feb 26 13:52:10 shakti kernel: [89127.982223]  [<c014be14>] getnstimeofday+0x34/0xf0
Feb 26 13:52:10 shakti kernel: [89127.982406]  [<c011c27c>] lapic_next_event+0xc/0x10
Feb 26 13:52:10 shakti kernel: [89127.982572]  [<c014e283>] clockevents_program_event+0xa3/0x150
Feb 26 13:52:10 shakti kernel: [89127.982855]  [<c021dd47>] rb_insert_color+0xb7/0xe0
Feb 26 13:52:10 shakti kernel: [89127.983063]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.983548]  [<f88e88b8>] handle_stripe+0x28/0xf00 [raid456]
Feb 26 13:52:10 shakti kernel: [89127.983967]  [<c0136a32>] __do_softirq+0x82/0x110
Feb 26 13:52:10 shakti kernel: [89127.984185]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.984690]  [<c0128cbe>] __wake_up+0x3e/0x60
Feb 26 13:52:10 shakti kernel: [89127.984912]  [<f88e3ace>] __release_stripe+0x10e/0x150 [raid456]
Feb 26 13:52:10 shakti kernel: [89127.985269]  [<f88e3b3f>] release_stripe+0x2f/0x50 [raid456]
Feb 26 13:52:10 shakti kernel: [89127.985611]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.986095]  [<f88e9b65>] raid5d+0x3d5/0x460 [raid456]
Feb 26 13:52:10 shakti kernel: [89127.986500]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.987005]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.987478]  [<f88b2ec3>] md_thread+0x23/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.987957]  [<c0145f20>] autoremove_wake_function+0x0/0x40
Feb 26 13:52:10 shakti kernel: [89127.988176]  [<f88b2ea0>] md_thread+0x0/0xe0 [md_mod]
Feb 26 13:52:10 shakti kernel: [89127.988650]  [<c0145c62>] kthread+0x42/0x70
Feb 26 13:52:10 shakti kernel: [89127.988802]  [<c0145c20>] kthread+0x0/0x70
Feb 26 13:52:10 shakti kernel: [89127.988968]  [<c010900b>] kernel_thread_helper+0x7/0x10
Feb 26 13:52:10 shakti kernel: [89127.989182]  =======================