netdev - Re: [Bugme-new] [Bug 12877] New: tg3: eth0 transit timed out, resetting -> dead NIC

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <49C82AE2.3080206@birkenwald.de>
Date:	Tue, 24 Mar 2009 01:35:46 +0100
From:	Bernhard Schmidt <berni@...kenwald.de>
To:	Matt Carlson <mcarlson@...adcom.com>
CC:	Michael Chan <mchan@...adcom.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"bugme-daemon@...zilla.kernel.org" <bugme-daemon@...zilla.kernel.org>
Subject: Re: [Bugme-new] [Bug 12877] New: tg3: eth0 transit timed out, resetting
 -> dead NIC

On 23.03.2009 19:18, Matt Carlson wrote:

Hello Matt,

>> Mar 22 04:06:46 svr02 kernel: [1392136.468921] PCI Memory Mapped IO Disabled!!!!
[...]
>> Mar 22 04:07:14 svr02 kernel: [1392164.768266] PCI Memory Mapped IO Disabled!!!!
>> at this point the "watchdog" kicked in and did rmmod/modprobe, so I
>> think the only thing you can read out of this debugging log is that
>> there was no kernel message right before MMIO got disabled and it takes
>> quite a while to fire the Tx timeout.
> So traffic on this box must be pretty light for the watchdog to fire off
> 30 seconds after the MMIO problem was detected, right?  Interesting.

Just to make sure I didn't confuse you, the "watchdog" I was talking 
about here is a shellscript like this, executed every minute

---
/bin/ping -q -c 5 <defaultgw> > /dev/null
RC=$?
if [ ${RC} -ne 0 ]; then
	rmmod tg3; sleep 5; modprobe tg3; sleep 5; ifup --force eth0
fi
---

at :46 MMIO was disabled, at :00 the cronjob started which took until 
:15 before detecting the network was dead and reloaded the modules

>> Mar 22 04:07:15 svr02 kernel: [1392165.540078] tg3 0000:03:04.1: PCI INT B disabled
>> Mar 22 04:07:16 svr02 kernel: [1392166.817125] tg3: tg3_abort_hw timed out for eth0, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
>> Mar 22 04:07:18 svr02 kernel: [1392168.398844] tg3: eth0: No firmware running.
>> Mar 22 04:07:29 svr02 kernel: [1392179.793309] tg3: eth0: Link is down.
>> Mar 22 04:07:31 svr02 kernel: [1392181.896030] tg3 0000:03:04.0: PCI INT A disabled
>> Mar 22 04:07:33 svr02 kernel: [1392183.957132] tg3.c:v3.94 (August 14, 2008)
>> Mar 22 04:07:33 svr02 kernel: [1392184.020034] tg3 0000:03:04.0: enabling device (0000 ->  0002)
>> Mar 22 04:07:33 svr02 kernel: [1392184.086083] tg3 0000:03:04.0: PCI INT A ->  GSI 16 (level, low) ->  IRQ 16

The tg3 watchdog (tg3: eth0: transmit timed out, resetting) did not 
appear at all in this circle, so I guess the checkscript killed the 
module before.

Yes, the NIC is very lightly loaded, around 100kbps / 70pps in each 
direction with a few occasional spikes.

>> I'm now switching to eth1.
> O.K.  I eagerly await your results.

So far so good, but it has only been running ~36 hours, that's not 
really a stability spree yet :-)

I'll keep you updated.

Bernhard
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html