netdev - RE: bnx2x - occasional high packet loss (on LAN)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 16 Sep 2015 08:15:41 +0000
From:	Ariel Elior <Ariel.Elior@...gic.com>
To:	Nikola Ciprich <nikola.ciprich@...uxbox.cz>,
	netdev <netdev@...r.kernel.org>
CC:	"nik@...uxbox.cz" <nik@...uxbox.cz>
Subject: RE: bnx2x - occasional high packet loss (on LAN)

Hi Nikola,
Please provide dmesg output from your system.
Thanks,
Ariel

> -----Original Message-----
> From: netdev-owner@...r.kernel.org [mailto:netdev-owner@...r.kernel.org] On
> Behalf Of Nikola Ciprich
> Sent: Tuesday, September 15, 2015 7:17 AM
> To: netdev <netdev@...r.kernel.org>
> Cc: nik@...uxbox.cz
> Subject: bnx2x - occasional high packet loss (on LAN)
> 
> Hello,
> 
> I'm trying to track strange issue with one of our servers and
> like to ask for recommendations..
> 
> I've got three node cluster (nodes A..C) interconnected with stacked broadcom
> ICX6610. eth0 of each box is connected to first switch, eth1 to second one,
> bonding set as follows: "mode=802.3ad lacp_rate=fast xmit_hash_policy=layer2+3
> miimon=100"
> 
> It happened few times, that suddenly eth1 on box A started misbehaving and
> communication
> with other nodes (ie flood ping) started dropping up to 30% packets. When this port
> has been shut on both sides, problem immediately vanished.
> 
> We've tried replacing card, cable and using different port on switch, but problem
> repeated again yesterday..
> 
> Since it's "only" loss, and not link loss, bonding doesn't help me much..
> 
> however during weekend, port also had strange link issue:
> 
> Sep 12 15:23:45 remrprv1a kernel: [676373.296786] bnx2x 0000:03:00.1 eth1: NIC
> Link is Down
> Sep 12 15:23:46 remrprv1a kernel: [676373.356638] bond0: link status definitely
> down for interface eth1, disabling it
> Sep 12 15:23:46 remrprv1a kernel: [676374.299571] bnx2x 0000:03:00.1 eth1: NIC
> Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
> Sep 12 15:23:47 remrprv1a kernel: [676374.364428] bond0: link status definitely up
> for interface eth1, 10000 Mbps full duplex
> Sep 12 15:23:47 remrprv1a kernel: [676374.372902] bond0: first active interface up!
> Sep 12 15:24:24 remrprv1a kernel: [676411.402511] bnx2x 0000:03:00.1 eth1: NIC
> Link is Down
> Sep 12 15:24:24 remrprv1a kernel: [676411.407422] bond0: link status definitely
> down for interface eth1, disabling it
> Sep 12 15:24:25 remrprv1a kernel: [676412.405311] bnx2x 0000:03:00.1 eth1: NIC
> Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
> Sep 12 15:24:25 remrprv1a kernel: [676412.408123] bond0: link status definitely up
> for interface eth1, 0 Mbps full duplex
> Sep 12 15:24:51 remrprv1a kernel: [676438.477641] bnx2x 0000:03:00.1 eth1: NIC
> Link is Down
> Sep 12 15:24:51 remrprv1a kernel: [676438.528513] bond0: link status definitely
> down for interface eth1, disabling it
> Sep 12 15:24:52 remrprv1a kernel: [676439.480472] bnx2x 0000:03:00.1 eth1: NIC
> Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
> Sep 12 15:24:52 remrprv1a kernel: [676439.536282] bond0: link status definitely up
> for interface eth1, 10000 Mbps full duplex
> 
> 0mbps link speed is quite weird I guess..
> 
> all three boxes are the same, running centos6 based system, 4.0.5 x86_64 kernel.
> 
> The only difference I noticed on them is, that irqbalance was enabled on problematic
> box and not on the others.. So I disabled it and rebooted the box.. The problem is,
> I can't really wait for the problem to reappear, so I'd like to ask, has anybody
> seen similar problem? I of so, was it fixed in some newer kernel release? I haven't
> found mention in the changelogs, but still.. or does somebody have a hint on what
> else
> I should check?
> 
> I'll try to reproduce this on test system (enabling irqbalance and doing some network
> benchmarks, but I'd be most happy if I could prevent it on this production system..)
> 
> thanks a lot for any advance
> 
> with best regards
> 
> nikola ciprich
> 
> PS: here's lspci -vv of eths.. should I provide any further information, please let me
> know:
> 
> http://nik.lbox.cz/download/lspci.txt
> 
> --
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28.rijna 168, 709 00 Ostrava
> 
> tel.:   +420 591 166 214
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@...uxbox.cz
> -------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html