lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <201005151026.20356.carsten.aulbert@aei.mpg.de>
Date:	Sat, 15 May 2010 10:26:17 +0200
From:	Carsten Aulbert <carsten.aulbert@....mpg.de>
To:	"\"Brandeburg, Jesse\"" <jesse.brandeburg@...el.com>
Cc:	Tejun Heo <tj@...nel.org>,
	"Ronciak, John" <john.ronciak@...el.com>,
	Henning Fehrmann <henning.fehrmann@....mpg.de>,
	"Kirsher, Jeffrey T" <jeffrey.t.kirsher@...el.com>,
	"Allan, Bruce W" <bruce.w.allan@...el.com>,
	"Waskiewicz Jr, Peter P" <peter.p.waskiewicz.jr@...el.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	Matt Mackall <mpm@...enic.com>,
	e1000-devel@...ts.sourceforge.net
Subject: Re: loosing IPMI-card by loading netconsole

Hi all,

I'll try to gather some information:

The system under question is Supermicro PSDML-LN2+ based, eth0 is a 82573E 
while eth1 is a 82573L. IPMI is piggy-bagged onto eth0, eth1 is our "data 
network" running with large jumbo frames:

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:30:48:96:e1:e2
          inet addr:172.26.1.26  Bcast:172.31.255.255  Mask:255.240.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:673128 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104281 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:46271457 (44.1 MiB)  TX bytes:8238586 (7.8 MiB)
          Memory:ee100000-ee120000

eth1      Link encap:Ethernet  HWaddr 00:30:48:96:e1:e3
          inet addr:10.10.1.26  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:17660268 errors:0 dropped:72239 overruns:0 frame:0
          TX packets:10941117 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:86507558735 (80.5 GiB)  TX bytes:7011613344 (6.5 GiB)
          Memory:ee200000-ee220000



# ethtool -i eth0
driver: e1000e
version: 1.0.2-k2
firmware-version: 0.15-4
bus-info: 0000:0d:00.0

ethtool -i eth1
driver: e1000e
version: 1.0.2-k2
firmware-version: 0.5-7
bus-info: 0000:0e:00.0

lscpi:
0d:00.0 0200: 8086:108c (rev 03)
        Subsystem: 15d9:108c    
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx- 
        Latency: 0, Cache Line Size: 64 bytes                                                                
        Interrupt: pin A routed to IRQ 29                                                                    
        Region 0: Memory at ee100000 (32-bit, non-prefetchable) [size=128K]                                  
        Region 2: I/O ports at 4000 [size=32]                                                                
        Capabilities: [c8] Power Management version 2                                                        
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA 
PME(D0+,D1-,D2-,D3hot+,D3cold+)                   
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-                                                  
        Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/0 Enable+                      
                Address: 00000000fee0f00c  Data: 41c1                                                        
        Capabilities: [e0] Express (v1) Endpoint, MSI 00                                                     
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, 
L1 <64us                      
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-                                      
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ 
Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ 
TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM unknown, 
Latency L0 <128ns, L1 <64us
                        ClockPM- Suprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Device Serial Number e2-e1-96-ff-ff-48-30-00
        Kernel driver in use: e1000e
        Kernel modules: e1000e

0e:00.0 0200: 8086:109a
        Subsystem: 15d9:109a
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 30
        Region 0: Memory at ee200000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at 5000 [size=32]
        Capabilities: [c8] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA 
PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/0 Enable+
                Address: 00000000fee0f00c  Data: 41b1
        Capabilities: [e0] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, 
L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ 
Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ 
TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM unknown, 
Latency L0 <128ns, L1 <64us
                        ClockPM+ Suprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Device Serial Number e3-e1-96-ff-ff-48-30-00
        Kernel driver in use: e1000e
        Kernel modules: e1000e


***************************************************************************
OK, now the detailed parts:

IPMI is currently blocked, i.e. I can query the card locally via the kernel 
module, but not remotely. netconsole is loaded and working via eth1.

# ethtool -S eth0                                     
NIC statistics:                                                                 
     rx_packets: 673400                                                         
     tx_packets: 104323                                                         
     rx_bytes: 48982938                                                         
     tx_bytes: 8659194                                                          
     rx_broadcast: 619125                                                       
     tx_broadcast: 3                                                            
     rx_multicast: 2088                                                         
     tx_multicast: 0                                                            
     rx_errors: 0                                                               
     tx_errors: 0                                                               
     tx_dropped: 0                                                              
     multicast: 2088                                                            
     collisions: 0                                                              
     rx_length_errors: 0                                                        
     rx_over_errors: 0                                                          
     rx_crc_errors: 0                                                           
     rx_frame_errors: 0                                                         
     rx_no_buffer_count: 0                                                      
     rx_missed_errors: 0                                                        
     tx_aborted_errors: 0                                                       
     tx_carrier_errors: 0                                                       
     tx_fifo_errors: 0                                                          
     tx_heartbeat_errors: 0                                                     
     tx_window_errors: 0                                                        
     tx_abort_late_coll: 0                                                      
     tx_deferred_ok: 0                                                          
     tx_single_coll_ok: 0                                                       
     tx_multi_coll_ok: 0                                                        
     tx_timeout_count: 0                                                        
     tx_restart_queue: 0                                                        
     rx_long_length_errors: 0                                                   
     rx_short_length_errors: 0                                                  
     rx_align_errors: 0                                                         
     tx_tcp_seg_good: 0                                                         
     tx_tcp_seg_failed: 0                                                       
     rx_flow_control_xon: 0                                                     
     rx_flow_control_xoff: 0                                                    
     tx_flow_control_xon: 0                                                     
     tx_flow_control_xoff: 0                                                    
     rx_long_byte_count: 48982938                                               
     rx_csum_offload_good: 76752                                                
     rx_csum_offload_errors: 0                                                  
     rx_header_split: 0                                                         
     alloc_rx_buff_failed: 0                                                    
     tx_smbus: 2                                                                
     rx_smbus: 594496                                                           
     dropped_smbus: 0                                                           
     rx_dma_failed: 0                                                           
     tx_dma_failed: 0                                                           
n0126:/build/ethregs-1.7.2# ethtool -S eth1
NIC statistics:                            
     rx_packets: 17664653                  
     tx_packets: 10941353                  
     rx_bytes: 86578358008                 
     tx_bytes: 7055320722                  
     rx_broadcast: 4761580                 
     tx_broadcast: 1252
     rx_multicast: 0
     tx_multicast: 0
     rx_errors: 0
     tx_errors: 0
     tx_dropped: 0
     multicast: 0
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_no_buffer_count: 1985
     rx_missed_errors: 72239
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     tx_restart_queue: 1920
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 127295
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_long_byte_count: 86578358008
     rx_csum_offload_good: 12907089
     rx_csum_offload_errors: 0
     rx_header_split: 750356
     alloc_rx_buff_failed: 0
     tx_smbus: 0
     rx_smbus: 0
     dropped_smbus: 0
     rx_dma_failed: 0
     tx_dma_failed: 0

ethreg result attached

******************************************************************************

After rebooting without netconsole the IPMI card was still not working, using 
the echo 1 > ... /remove command from Henning's original email suddenly made 
IPMI working again.

After that I rebooted the machine while watching the progress via ipmitool's 
sol activate, however during the boot sequence suddenly the kernel output 
slowed to a crawl. About 1-2 minutes later remote IPMI broke down again.

netconsole was NOT loaded, thus I don't think that this is really related.

OK, how to proceed?

e.g. how can I re-enable the interface once I issued

echo 1 > /sys/devices/pci0000:00/0000:00:1c.4/0000:0d:00.0/remove

Cheers

Carsten

View attachment "ethregs.IPMInotworkingNOnetconsole" of type "text/plain" (41597 bytes)

View attachment "ethregs.IPMInotworking" of type "text/plain" (41598 bytes)

Powered by blists - more mailing lists