netdev - Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20231128021958.GA93203@dev-dsk-ipman-2a-ee5dfd20.us-west-2.amazon.com>
Date: Tue, 28 Nov 2023 02:20:03 +0000
From: Ivan Pang <ipman@...zon.com>
To: Skyler Mäntysaari <sm+lists@...m.fi>
CC: Jesse Brandeburg <jesse.brandeburg@...el.com>, <netdev@...r.kernel.org>,
	<intel-wired-lan@...ts.osuosl.org>, Jordan Crouse <jorcrous@...zon.com>
Subject: Re: [Intel-wired-lan] The difference between linux kernel driver and
 FreeBSD's with Intel X533

On Wed, Oct 18, 2023 at 09:50:35PM +0300, Skyler Mäntysaari wrote:
> On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
> > On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
> >> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
> >>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
> >>>>>> Hi there,
> >>>>>>
> >>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
> >>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
> >>>>>> it does in OPNsense which is based on FreeBSD 13.2.
> >>>>>>
> >>>>>> How would I go about debugging this properly? Both sides see light,
> >>>>>> but no link unless I'm using FreeBSD.
> >>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
> >>>
> >>> Hi Skyler,
> >>>
> >>> Response from Intel team:
> >>>
> >>> In the ethtool -m output pasted I see TX and RX optical power is fine.
> >>> Could you request fault status on both sides? Just want to check if link
> >>> is down because we are at local-fault or link partner is at local-fault.
> >>>
> >>> rmmod ixgbe
> >>> modprobe ixgbe
> >>> ethtool -S eth0 | grep fault
> >>>
> >>> Since it is 10G, if our side TX is ON (power level says it is) then we
> >>> should expect link partner RX to be locked so cannot be at Local Fault.
> >>>
> >>> Skyler, please gather that ethtool -S data for us.
> >>>
> >>> Jesse
> >>>
> >>>
> >>>
> >>> 
> >>
> >> Hi Jesse,
> >>
> >> As the other side of the link is an Juniper, I'm not quite sure how I 
> >> would gather the same data from it as it doesn't have ethtool?
> >>
> >> I have also somewhat given up hope on it working on VyOS and instead I 
> >> am using OPNsense for the moment but I still have VyOS installed as 
> >> well.
> >>
> >> Skyler
> >
> > Hi Jesse,
> >
> > I did verify that the grep doesn't yield any results on the VyOS box 
> > and all of the data returned has an value of 0. Paste of which is here: 
> > https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
> >
> > I really think something weird is going on with the driver in Linux as 
> > otherwise the same exact config on Juniper wouldn't work there either. 
> > The VyOS box also says that it's unable to modify autoneg settings, or 
> > speed/duplex of the interface.
> >
> > Skyler
> 
> It has been  verified that the driver in kernel version 5.4.255 seems to work aka 1.3 VyOS.  Post from another user in the same thread about it: https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46
> 
> I have also verified that the out-of-tree ixgbe driver does work, but in-kernel doesn't in kernel 6.1.58.
> 
> Please share these findings with the correct Intel team so that this could be fixed.
> 
> - Skyler
> 

Hi Skyler, Jesse,

I came across this very similar issue when upgrading our networking gear
from kernel 5.15 to 6.1. Our 10G link fails with the in-tree 6.1 ixgbe
driver but works with the out-of-tree 5.x versions. I found that my link
issues were related to this commit:

ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.1.63&id=565736048bd5f9888990569993c6b6bfdf6dcb6d

Specifically, our 10G link works when both sides of the fiber are
running the in-tree 6.1 ixgbe driver with this autonegotiation change.
Our link also works when both sides are running the 5.x ixgbe drivers
without this commit. It fails, however, when only one side has this
commit. Our current setup compiles the in-tree 6.1 ixgbe driver with
this commit reverted, for compatibility with our varying hardware.

I would appreciate it if anyone can cross-check my claim with their
hardware as well. Also, would anyone be able to help explain what some
of those registers and reg_val being written are doing?

-Ivan