netdev - Re: bnx2x - odd behaviour

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Thu, 11 Apr 2019 10:56:28 +0200
From:   Ian Kumlien <ian.kumlien@...il.com>
To:     Sudarsana Reddy Kalluru <skalluru@...vell.com>
Cc:     Linux Kernel Network Developers <netdev@...r.kernel.org>,
        Ariel Elior <aelior@...vell.com>
Subject: Re: bnx2x - odd behaviour

On Thu, Apr 4, 2019 at 4:27 PM Sudarsana Reddy Kalluru
<skalluru@...vell.com> wrote:
>
> Hi,
>    We are not aware of this issue. Please collect the register dump i.e., "ethtool -d <interface>" output when this issue happens (before performing link-flap) and share it for the analysis.

I haven't been able to recreate the original issue, but i just had
something completely new happen that might be related.

FYI, this is old HP blade servers using a pass-trough module (and they
can be dodgy at times)...

I brought up the second nic to enable network redundancy and the
machine crashed (could only see the tail of it as is)
but the interesting bit is that it wouldn't boot properly - resulting
in the picture below

https://photos.app.goo.gl/pyKEnu9qLLfvGeXC6

I don't know how useful this is, if at all, but it does seem like it
is in a incorrect state - a cold boot fixed it.

Still trying to recreate the original issue....

> Thanks,
> Sudarsana
> > -----Original Message-----
> > From: netdev-owner@...r.kernel.org <netdev-owner@...r.kernel.org> On
> > Behalf Of Ian Kumlien
> > Sent: Wednesday, April 3, 2019 8:31 PM
> > To: Linux Kernel Network Developers <netdev@...r.kernel.org>; Ariel Elior
> > <aelior@...vell.com>; Sudarsana Reddy Kalluru <skalluru@...vell.com>
> > Subject: bnx2x - odd behaviour
> >
> > Hi,
> >
> > We just had this happen on 5.0.2
> >
> > It looks like the interface went down, ended up in a broken state and a ip li
> > set down/up dev enp2s0f0 made it work again
> >
> > It looks really weird and I haven't really seen anything like it, anyone with a
> > clue?
> >
> > dmesg:
> > ....
> > [1310361.808694] bnx2x 0000:02:00.0 enp2s0f0: NIC Link is Down
> > [1310361.824554] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1310362.872678] bond0: link status definitely down
> > for interface enp2s0f0, disabling it [1310362.880691] device enp2s0f0 left
> > promiscuous mode [1310363.188592] bnx2x 0000:02:00.0 enp2s0f0: NIC Link
> > is Up, 10000 Mbps full duplex, Flow control: ON - transmit [1310363.200653]
> > bond0: link status up for interface enp2s0f0, enabling it in 0 ms
> > [1310363.208192] bond0: link status definitely up for interface enp2s0f0,
> > 10000 Mbps full duplex [1310363.216885] bond0: making interface enp2s0f0
> > the new active one [1310363.223075] device enp2s0f0 entered promiscuous
> > mode [1310363.228613] bond0: first active interface up!
> > [1310364.048805] bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1310364.058297] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> > [1310365.072604] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1) [1310366.096679] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (2) [1310366.103922]
> > bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1310366.113387] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> > [1310367.120518] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (3) [1310368.144635] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (4) [1310369.168591]
> > bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (5)
> > [1310371.216519] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (6) ... it does go on ...
> > [1312156.028230] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1520) [1312157.052226] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (1521)
> > [1312157.059842] bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1312157.069242] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> > [1312158.076261] bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1312158.085657] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> > [1312159.100154] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1522) [1312160.124226] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (1523)
> > [1312161.148127] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1524) [1312162.172102] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (1525)
> > [1312163.196000] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1526) [1312163.203610] bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1312163.213082] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> > [1312164.220248] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1527) [1312165.244119] bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1312165.253524] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> > [1312166.268053] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1528) [1312167.292105] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (1529)
> > [1312168.316022] bnx2x: [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer
> > max (1530) [1312169.340014] bnx2x:
> > [bnx2x_hw_stats_update:869(enp2s0f0)]NIG timer max (1531)
> > [1312169.347584] bnx2x:
> > [bnx2x_attn_int_deasserted3:4357(enp2s0f0)]LATCHED attention
> > 0x04000000 (masked)
> > [1312169.357054] bnx2x: [bnx2x_attn_int_deasserted3:4361(enp2s0f0)]GRC
> > time-out 0x08004384
> >
> > ... trying to bing it down ...
> >
> > [1312169.659992] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1312169.672041] bond0: link status down for active
> > interface enp2s0f0, disabling it in 1000 ms [1312169.682084] bond0: link
> > status down for active interface enp2s0f0, disabling it in 1000 ms
> > [1312169.692159] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1312169.702026] bond0: link status down for active
> > interface enp2s0f0, disabling it in 1000 ms [1312169.712081] bond0: link
> > status down for active interface enp2s0f0, disabling it in 1000 ms
> > [1312169.722097] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1312169.732073] bond0: link status down for active
> > interface enp2s0f0, disabling it in 1000 ms [1312169.742079] bond0: link
> > status down for active interface enp2s0f0, disabling it in 1000 ms
> > [1312169.752066] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1312169.762017] bond0: link status down for active
> > interface enp2s0f0, disabling it in 1000 ms [1312169.771958] bond0: link
> > status down for active interface enp2s0f0, disabling it in 1000 ms
> > [1312169.782085] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms .... on and on ...
> > [1312170.434045] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1312170.444012] bond0: link status down for active
> > interface enp2s0f0, disabling it in 1000 ms [1312170.454024] bond0: link
> > status down for active interface enp2s0f0, disabling it in 1000 ms
> > [1312170.463879] bond0: link status down for active interface enp2s0f0,
> > disabling it in 1000 ms [1312170.473950] bond0: link status down for active
> > interface enp2s0f0, disabling it in 1000 ms [1312170.484107] bond0: link
> > status down for active interface enp2s0f0, disabling it in 1000 ms
> > [1312171.532119] bond0: link status definitely down for interface enp2s0f0,
> > disabling it
> >
> > ... bringing it up again ...
> >
> > [1312171.540128] device enp2s0f0 left promiscuous mode [1312189.213375]
> > bnx2x 0000:02:00.0 enp2s0f0: using MSI-X  IRQs: sp 42  fp[0] 44 ... fp[7] 51
> > [1312190.780919] bnx2x 0000:02:00.0 enp2s0f0: NIC Link is Up, 10000 Mbps
> > full duplex, Flow control: ON - transmit [1312190.787840] bond0: link status
> > up for interface enp2s0f0, enabling it in 0 ms [1312190.798618] bond0: link
> > status definitely up for interface enp2s0f0, 10000 Mbps full duplex
> > [1312190.807307] bond0: making interface enp2s0f0 the new active one
> > [1312190.813560] device enp2s0f0 entered promiscuous mode
> > [1312190.820884] bond0: first active interface up!
> > ---