netdev - Re: sky2 hangs without any messages

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20070712144334.0bc02b17@freepuppy.rosehill.hemminger.net>
Date:	Thu, 12 Jul 2007 14:43:34 -0700
From:	Stephen Hemminger <shemminger@...ux-foundation.org>
To:	"Daniel J Blueman" <daniel.blueman@...il.com>
Cc:	"Linux Netdev" <netdev@...r.kernel.org>
Subject: Re: sky2 hangs without any messages

On Thu, 12 Jul 2007 22:29:50 +0100
"Daniel J Blueman" <daniel.blueman@...il.com> wrote:

> On 12/07/07, Stephen Hemminger <shemminger@...ux-foundation.org> wrote:
> > On Wed, 11 Jul 2007 23:55:29 +0100
> > "Daniel J Blueman" <daniel.blueman@...il.com> wrote:
> >
> > > > >
> > > > > Please try again with post 2.6.22 git version (1.16)?
> > > >
> > > > Reproduced with 2.6.22 w/ sky2 1.16 from git. We observe this
> > > > characteristic failure on the NFS server (always around 2-3GB of
> > > > transmit):
> > > >
> > > > $ ifconfig lan0
> > > > lan0      Link encap:Ethernet  HWaddr 00:03:2D:05:9C:27
> > > >           inet addr:192.168.0.250  Bcast:192.168.0.255  Mask:255.255.255.0
> > > >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:24007220 errors:1 dropped:1 overruns:0 frame:1
> > > >           TX packets:13886495 errors:0 dropped:0 overruns:0 carrier:0
> > > >           collisions:0 txqueuelen:1000
> > > >           RX bytes:171026170 (163.1 MiB)  TX bytes:2262910580 (2.1 GiB)
> > > >           Interrupt:16
> > > >
> > > > I'll rebuild with debugfs and grab the debug you've exported.
> > >
> > > In quiescent state [1] and in failure state [2]. This time, 2 framing
> > > failures [3]; took 3.6GB of transmit to hit the window.
> >
> > Since the IRQ workaround has a timeout of 100ms. I observed cases where
> > the TCP connection dropped (because of lost packets), but the network device
> > then recovered.  Can you ping the other side after it hangs?  Or reconnect?
> >
> > Ifconfig lumps a bunch of different errors together so it can confuse the issue.
> > Preference is for:
> >         ip -s -s link show eth0
> > or
> >         grep -v '^0' /sys/class/net/eth0/statistics/*
> >
> > If the framing error does reproduce with the hang, perhaps the chip needs some
> > receive flush logic to recover. Receive errors normally put a message in syslog
> > output, did you look there?
> >
> > > Daniel
> > >
> > > --- [1]
> > >
> > > # cat sky2/lan0
> > > IRQ src=0 mask=c000001d control=0
> > > Status ring (empty)
> > > Tx ring pending=191...191 report=191 done=191
> > >
> > > Rx ring hw get=956 put=61 last=1023
> > >
> > > --- [2]
> > >
> > > # cat sky2/lan0
> > > IRQ src=0 mask=c000001d control=0
> > > Status ring (empty)
> > > Tx ring pending=251...251 report=251 done=251
> > >
> > > Rx ring hw get=1020 put=160 last=1023
> > >
> > > --- [3]
> > >
> > > $ ifconfig lan0
> > > lan0      Link encap:Ethernet  HWaddr 00:03:2D:05:9C:27
> > >           inet addr:192.168.0.250  Bcast:192.168.0.255  Mask:255.255.255.0
> > >           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> > >           RX packets:13304841 errors:1 dropped:1 overruns:0 frame:2
> > >           TX packets:7493765 errors:0 dropped:0 overruns:0 carrier:0
> > >           collisions:0 txqueuelen:1000
> > >           RX bytes:232720755 (221.9 MiB)  TX bytes:3964088142 (3.6 GiB)
> > >           Interrupt:16
> >
> > You aren't hung because of lost IRQ. When than happens the debugfs output will have
> > a bunch of Tx packets stuck (not cleaned up), and Status messages, and receive packets.
> 
> I'll grab the above info when I next get chance.
> 
> The vendor driver recovery process may be worthwhile taking a look at;
> I guess you've seen the code near the bottom of skge.c (under 'case
> SK_DRV_RECOVER')? The driver kicks the chip with
> SK_PNMI_EVT_XMAC_RESET and calls SkYuk2RestartRxBmu - perhaps
> something like this sequence is needed for a more targetted approach?

That code triggers (falsely) on an idle or barely active link. It won't work.
It is covering over a bunch of problems in the vendor driver that like improper
flow control.

-- 
Stephen Hemminger <shemminger@...ux-foundation.org>
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html