20110822.log:Aug 22 03:13:33 console1 port05 RXDATA: [10780.415059] CE: hpet increased min_delta_ns to 20113 nsec 20110823.log:Aug 23 02:50:54 console1 port05 RXDATA: [95603.710373] CE: hpet increased min_delta_ns to 30169 nsec 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.546556] ------------[ cut here ]------------ 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.551315] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.559221] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.563617] Watchdog detected hard LOCKUP on cpu 0 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.568364] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.603809] Pid: 2416, comm: flush-8:0 Not tainted 3.1.0-rc2+ #2 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.609994] Call Trace: 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.612549] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.619333] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.625220] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.631897] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.638309] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.643845] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.650693] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.656755] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.663115] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.669439] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.675499] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.682173] [] notify_die+0x2e/0x30 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.687472] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.692481] [] nmi+0x20/0x30 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.697165] [] ? cpu_relax+0xb/0xd 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.702389] <> [] try_to_wake_up+0x89/0x1ca 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.709536] [] ? wake_up_process+0x15/0x17 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.715422] [] default_wake_function+0x12/0x14 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.721659] [] __wake_up_common+0x4e/0x84 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.727489] [] ? _xfs_buf_ioend+0x2f/0x36 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.733852] [] complete+0x3f/0x52 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.738985] [] xfs_buf_ioend+0xc2/0xe2 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.745079] [] _xfs_buf_ioend+0x2f/0x36 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.751259] [] xfs_buf_bio_end_io+0x2a/0x37 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.757758] [] bio_endio+0x2d/0x2f 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.762945] [] req_bio_endio.clone.31+0x9e/0xa7 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.772811] [] blk_update_request+0x191/0x32e 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.782563] [] blk_update_bidi_request+0x21/0x6d 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.792529] [] blk_end_bidi_request+0x1f/0x5d 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.802230] [] blk_end_request+0x10/0x12 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.811446] [] scsi_io_completion+0x1e2/0x4d8 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.821002] [] ? _raw_spin_unlock_irqrestore+0x17/0x19 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.831405] [] scsi_finish_command+0xe8/0xf1 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.840948] [] scsi_softirq_done+0x109/0x112 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.850485] [] blk_done_softirq+0x73/0x87 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.859729] [] __do_softirq+0xc9/0x1b6 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.868705] [] ? paravirt_read_tsc+0x9/0xd 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.878093] [] call_softirq+0x1c/0x30 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.887025] [] do_softirq+0x46/0x84 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.895767] [] irq_exit+0x52/0xaf 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.904246] [] smp_apic_timer_interrupt+0x7c/0x8a 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.914171] [] apic_timer_interrupt+0x6e/0x80 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.923618] [] ? arch_local_irq_restore+0x6/0xd 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.933756] [] _raw_spin_unlock_irqrestore+0x17/0x19 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.943630] [] hpsa_scsi_queue_command+0x2f9/0x319 [hpsa] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.953851] [] scsi_dispatch_cmd+0x18c/0x251 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.962917] [] scsi_request_fn+0x3cd/0x3f9 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.971781] [] __blk_run_queue+0x1b/0x1d 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.980439] [] cfq_insert_request+0x4cf/0x504 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.989469] [] __elv_add_request+0x19c/0x1e7 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145813.998481] [] blk_flush_plug_list+0x166/0x19b 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.007670] [] schedule+0x2a3/0x6dc 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.015903] [] schedule_timeout+0x36/0xe3 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.024644] [] ? submit_bio+0xe0/0xff 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.032972] [] ? bio_alloc_bioset+0x4d/0xc4 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.041868] [] ? need_resched+0x23/0x2d 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.050400] [] wait_for_common+0x9a/0xef 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.059015] [] ? try_to_wake_up+0x1ca/0x1ca 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.067925] [] ? _xfs_buf_read+0x43/0x48 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.077050] [] ? xfs_trans_read_buf+0x1df/0x31c [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.086775] [] wait_for_completion+0x1d/0x1f 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.095822] [] xfs_buf_iowait+0x43/0x91 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.104964] [] _xfs_buf_read+0x43/0x48 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.113940] [] xfs_buf_read+0x84/0xcd [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.122800] [] xfs_trans_read_buf+0x1df/0x31c [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.132355] [] xfs_btree_read_buf_block.clone.22+0x4f/0xa7 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.143044] [] ? kmem_zone_alloc+0x3d/0x87 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.152365] [] xfs_btree_lookup_get_block+0x84/0xac [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.162459] [] xfs_btree_lookup+0x12b/0x3dc [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.171911] [] ? kmem_zone_zalloc+0x24/0x35 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.181371] [] ? xfs_allocbt_init_cursor+0x3e/0xa7 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.191368] [] xfs_alloc_lookup_ge.clone.3+0x1d/0x1f [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.201594] [] xfs_alloc_ag_vextent_near+0x76/0x94f [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.211727] [] xfs_alloc_ag_vextent+0x33/0xcd [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.221327] [] xfs_alloc_vextent+0x2ad/0x478 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.230842] [] xfs_bmap_btalloc+0x35d/0x50a [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.240288] [] xfs_bmap_alloc+0xe/0x10 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.249333] [] xfs_bmapi+0x6d7/0xd5d [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.258164] [] ? _cond_resched+0xe/0x22 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.266721] [] ? slab_pre_alloc_hook.clone.37+0x1a/0x1e 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.276626] [] ? should_resched+0xe/0x2e 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.285264] [] ? xfs_trans_add_item+0x50/0x5c [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.294920] [] xfs_iomap_write_allocate+0x1c5/0x2d2 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.305074] [] xfs_map_blocks+0x161/0x1fa [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.314364] [] xfs_vm_writepage+0x219/0x3f0 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.323796] [] __writepage+0x17/0x30 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.332147] [] write_cache_pages+0x26a/0x3d1 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.341156] [] ? set_page_dirty+0x60/0x60 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.349892] [] generic_writepages+0x45/0x5c 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.358846] [] xfs_vm_writepages+0x4d/0x56 [xfs] 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.368206] [] do_writepages+0x21/0x2a 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.376634] [] writeback_single_inode+0x113/0x2c7 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.386011] [] writeback_sb_inodes+0x174/0x215 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.395106] [] __writeback_inodes_wb+0x78/0xb9 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.404153] [] wb_writeback+0x125/0x23d 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.412596] [] ? get_nr_inodes_unused+0x48/0x60 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.421829] [] wb_do_writeback+0x148/0x1b6 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.430633] [] bdi_writeback_thread+0x7b/0x21b 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.439736] [] ? wb_do_writeback+0x1b6/0x1b6 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.448701] [] kthread+0x82/0x8a 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.456626] [] kernel_thread_helper+0x4/0x10 20110823.log:Aug 23 16:49:53 console1 port05 RXDATA: [145814.465568] [] ? kthread_worker_fn+0x14b/0x14b 20110823.log:Aug 23 16:49:54 console1 port05 RXDATA: [145814.474730] [] ? gs_change+0x13/0x13 20110823.log:Aug 23 16:49:54 console1 port05 RXDATA: [145814.482982] ---[ end trace 178d61af15d42cc2 ]--- 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.435131] ------------[ cut here ]------------ 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.442748] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.453549] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.460868] Watchdog detected hard LOCKUP on cpu 6 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.465611] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.516885] Pid: 4637, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.527236] Call Trace: 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.533157] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.543400] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.552738] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.562919] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.572741] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.581677] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.591968] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.601453] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.611233] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.621081] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.630639] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.640778] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.649522] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.657990] [] nmi+0x20/0x30 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.666118] <> 20110823.log:Aug 23 17:08:00 console1 port05 RXDATA: [146898.668323] ---[ end trace 178d61af15d42cc3 ]--- 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146909.966143] ------------[ cut here ]------------ 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146909.974282] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146909.985645] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146909.993499] Watchdog detected hard LOCKUP on cpu 13 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146909.998330] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.052268] Pid: 26054, comm: httpd Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.062907] Call Trace: 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.069369] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.080124] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.089968] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.100696] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.111161] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.120711] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.131583] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.141669] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.152070] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.162431] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.172528] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.183238] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.192547] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.201515] [] nmi+0x20/0x30 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.210053] <> 20110823.log:Aug 23 17:08:12 console1 port05 RXDATA: [146910.212260] ---[ end trace 178d61af15d42cc4 ]--- 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.348076] ------------[ cut here ]------------ 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.356596] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.368111] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.376094] Watchdog detected hard LOCKUP on cpu 1 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.380837] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.434971] Pid: 2364, comm: auth Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.445462] Call Trace: 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.451981] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.462703] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.472547] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.483173] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.493591] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.503092] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.513867] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.523903] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.534217] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.544597] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.554718] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.565462] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.574767] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.583758] [] nmi+0x20/0x30 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.592339] <> 20110823.log:Aug 23 17:08:22 console1 port05 RXDATA: [146920.594546] ---[ end trace 178d61af15d42cc5 ]--- 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.350314] ------------[ cut here ]------------ 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.358745] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.370298] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.378327] Watchdog detected hard LOCKUP on cpu 9 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.383071] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.437130] Pid: 4628, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.448026] Call Trace: 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.454490] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.465266] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.475146] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.485786] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.496165] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.505627] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.516467] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.526554] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.536832] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.547173] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.557240] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.567986] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.577342] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.586384] [] nmi+0x20/0x30 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.595006] <> 20110823.log:Aug 23 17:08:32 console1 port05 RXDATA: [146930.597213] ---[ end trace 178d61af15d42cc6 ]--- 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.613819] ------------[ cut here ]------------ 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.622298] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.633903] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.641902] Watchdog detected hard LOCKUP on cpu 12 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.646746] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.700809] Pid: 4668, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.711709] Call Trace: 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.718203] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.728984] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:08:42 console1 port05 RXDATA: [146940.738884] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.749526] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.759873] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.769387] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.780257] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.790308] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.800678] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.811023] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.821111] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.831866] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.841201] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.850246] [] nmi+0x20/0x30 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.858886] <> 20110823.log:Aug 23 17:08:43 console1 port05 RXDATA: [146940.861090] ---[ end trace 178d61af15d42cc7 ]--- 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.406589] ------------[ cut here ]------------ 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.415077] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.426688] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.434688] Watchdog detected hard LOCKUP on cpu 15 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.439520] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.493547] Pid: 2688, comm: httpd Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.504178] Call Trace: 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.510705] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.521472] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.531320] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.541953] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.552290] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.561838] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.572706] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.582703] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.593018] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.603323] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.613443] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.624188] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.633495] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.642526] [] nmi+0x20/0x30 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.651151] <> 20110823.log:Aug 23 17:08:52 console1 port05 RXDATA: [146950.653358] ---[ end trace 178d61af15d42cc8 ]--- 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.767570] ------------[ cut here ]------------ 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.776089] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.787597] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.795582] Watchdog detected hard LOCKUP on cpu 7 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.800326] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.854416] Pid: 4702, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.865218] Call Trace: 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.871705] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.882390] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.892196] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.902839] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.913265] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.922816] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.933641] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.943681] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.953957] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.964338] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.974418] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.985160] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146959.994468] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146960.003416] [] nmi+0x20/0x30 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146960.012002] <> 20110823.log:Aug 23 17:09:02 console1 port05 RXDATA: [146960.014209] ---[ end trace 178d61af15d42cc9 ]--- 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.305541] ------------[ cut here ]------------ 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.313978] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.325543] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.333586] Watchdog detected hard LOCKUP on cpu 11 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.338417] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.392613] Pid: 4700, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.403474] Call Trace: 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.409979] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.420731] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.430557] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.441175] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.451580] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.461150] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.472038] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.482147] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.492487] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.502805] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.512946] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.523717] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.533054] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.542117] [] nmi+0x20/0x30 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.550772] <> 20110823.log:Aug 23 17:09:11 console1 port05 RXDATA: [146969.552978] ---[ end trace 178d61af15d42cca ]--- 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.786384] ------------[ cut here ]------------ 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.794886] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.806426] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.814443] Watchdog detected hard LOCKUP on cpu 5 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.819187] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.873347] Pid: 27, comm: ksoftirqd/5 Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.884341] Call Trace: 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.890853] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.901651] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.911481] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.922149] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.932599] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.942180] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.953074] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.963140] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.973485] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.983837] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146978.993984] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.004755] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.014132] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.023151] [] nmi+0x20/0x30 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.031765] [] ? read_tsc+0x1/0x19 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.040890] <> [] ? timekeeping_get_ns+0x1a/0x39 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.051713] [] ktime_get+0x5e/0x99 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.060663] [] tick_dev_program_event+0x25/0xf5 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.070717] [] tick_program_event+0x1f/0x21 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.080320] [] hrtimer_interrupt+0x109/0x197 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.089955] [] __hrtimer_peek_ahead_timers+0x37/0x39 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.100214] [] hrtimer_peek_ahead_timers+0x20/0x2b 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.110276] [] run_hrtimer_softirq+0xe/0x10 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.119704] [] __do_softirq+0xc9/0x1b6 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.128685] [] run_ksoftirqd+0x7f/0x11d 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.137729] [] ? __do_softirq+0x1b6/0x1b6 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.146935] [] kthread+0x82/0x8a 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.155339] [] kernel_thread_helper+0x4/0x10 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.164789] [] ? kthread_worker_fn+0x14b/0x14b 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.174457] [] ? gs_change+0x13/0x13 20110823.log:Aug 23 17:09:21 console1 port05 RXDATA: [146979.183250] ---[ end trace 178d61af15d42ccb ]--- 20110823.log:Aug 23 17:09:30 console1 port05 RXDATA: [146988.550621] ------------[ cut here ]------------ 20110823.log:Aug 23 17:09:30 console1 port05 RXDATA: [146988.558776] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:09:30 console1 port05 RXDATA: [146988.570148] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:09:30 console1 port05 RXDATA: [146988.578075] Watchdog detected hard LOCKUP on cpu 10 20110823.log:Aug 23 17:09:30 console1 port05 RXDATA: [146988.582909] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:09:30 console1 port05 RXDATA: [146988.637003] Pid: 4748, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.647881] Call Trace: 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.654404] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.665083] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.674792] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.685126] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.695122] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.704170] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.714595] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.724228] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.734054] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.743834] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.753384] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.763504] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.772219] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.780653] [] nmi+0x20/0x30 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.788715] <> 20110823.log:Aug 23 17:09:31 console1 port05 RXDATA: [146988.790929] ---[ end trace 178d61af15d42ccc ]--- 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.073402] ------------[ cut here ]------------ 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.081577] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.092881] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.100795] Watchdog detected hard LOCKUP on cpu 8 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.105539] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.159535] Pid: 4771, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.170342] Call Trace: 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.176836] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.187614] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.197432] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.208087] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.218528] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.228120] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.239044] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.249149] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.259519] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.269847] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.280005] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.290770] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.300078] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.309090] [] nmi+0x20/0x30 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.317691] <> 20110823.log:Aug 23 17:09:45 console1 port05 RXDATA: [147003.319897] ---[ end trace 178d61af15d42ccd ]--- 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.894083] ------------[ cut here ]------------ 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.902532] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.914153] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.922169] Watchdog detected hard LOCKUP on cpu 14 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.927001] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.981078] Pid: 4682, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.991987] Call Trace: 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147014.998492] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.009284] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.019198] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.029862] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.040266] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.049796] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.060684] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.070789] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.081132] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.091495] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.101578] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.112311] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.121687] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.130701] [] nmi+0x20/0x30 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.139308] <> 20110823.log:Aug 23 17:09:57 console1 port05 RXDATA: [147015.141531] ---[ end trace 178d61af15d42cce ]--- 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.268508] ------------[ cut here ]------------ 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.276969] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.288552] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.296523] Watchdog detected hard LOCKUP on cpu 3 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.301268] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.355381] Pid: 4676, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.366256] Call Trace: 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.372770] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.383527] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.393450] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.404162] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.414568] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.424058] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.434863] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.444979] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.455327] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.465736] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.475799] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.486525] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.495855] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.504844] [] nmi+0x20/0x30 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.513420] <> 20110823.log:Aug 23 17:10:06 console1 port05 RXDATA: [147024.515626] ---[ end trace 178d61af15d42ccf ]--- 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147033.906597] ------------[ cut here ]------------ 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147033.915046] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147033.926669] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147033.934685] Watchdog detected hard LOCKUP on cpu 2 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147033.939430] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147033.993555] Pid: 1993, comm: automount Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.004553] Call Trace: 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.011107] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.021906] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.031829] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.042491] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.052898] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.062390] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.073239] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.083355] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.093659] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.104023] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.114092] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.124866] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.134251] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.143269] [] nmi+0x20/0x30 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.151885] [] ? sys_futex+0x91/0x146 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.161324] <> [] system_call_fastpath+0x16/0x1b 20110823.log:Aug 23 17:10:16 console1 port05 RXDATA: [147034.172157] ---[ end trace 178d61af15d42cd0 ]--- 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.624844] ------------[ cut here ]------------ 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.633296] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9b/0xa6() 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.644875] Hardware name: ProLiant DL380 G6 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.652917] Watchdog detected hard LOCKUP on cpu 4 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.657661] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl sunrpc mptctl mptbase ipmi_devintf autofs4 8021q garp stp llc ipt_LOG iptable_mangle uinput ipmi_si pcspkr hpilo ipmi_msghandler hpwdt iTCO_wdt iTCO_vendor_support i7core_edac edac_core acpi_power_meter bnx2 serio_raw microcode xfs hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.711976] Pid: 4752, comm: sendmail Tainted: G W 3.1.0-rc2+ #2 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.722945] Call Trace: 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.729548] [] warn_slowpath_common+0x85/0x9d 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.740391] [] warn_slowpath_fmt+0x46/0x48 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.750360] [] watchdog_overflow_callback+0x9b/0xa6 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.761078] [] __perf_event_overflow+0x100/0x182 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.771529] [] ? local_clock+0x27/0x2f 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.781156] [] ? perf_event_update_userpage+0x11/0xa5 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.792097] [] perf_event_overflow+0x14/0x16 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.802280] [] intel_pmu_handle_irq+0x387/0x3e6 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.812696] [] perf_event_nmi_handler+0x46/0x91 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.823116] [] notifier_call_chain+0x37/0x63 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.833328] [] atomic_notifier_call_chain+0x18/0x1a 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.844157] [] notify_die+0x2e/0x30 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.853550] [] do_nmi+0x6f/0x218 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.862626] [] nmi+0x20/0x30 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.871332] <> 20110823.log:Aug 23 17:10:34 console1 port05 RXDATA: [147051.873537] ---[ end trace 178d61af15d42cd1 ]--- 20110823.log:Aug 23 17:18:30 console1 port05 RXDATA: [147526.880360] NMI: PCI system error (SERR) for reason a1 on CPU 0. 20110823.log:Aug 23 17:18:30 console1 port05 RXDATA: [147526.890196] Dazed and confused, but trying to continue 20110823.log:Aug 23 17:18:40 console1 portmanager[1635]: INFO portmanager - /dev/port05: DCD -> 0 20110823.log:Aug 23 17:18:40 console1 port05 SIGNAL: DCD=off