netdev - Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c229a29d-84a9-4390-a138-2034cd3bf572@skym.fi>
Date: Tue, 28 Nov 2023 15:58:44 +0200
From: Skyler Mäntysaari <sm+lists@...m.fi>
To: Jesse Brandeburg <jesse.brandeburg@...el.com>
Cc: Jordan Crouse <jorcrous@...zon.com>, netdev@...r.kernel.org,
 intel-wired-lan@...ts.osuosl.org
Subject: Re: [Intel-wired-lan] The difference between linux kernel driver and
 FreeBSD's with Intel X533


On 11/28/23 04:20, Ivan Pang wrote:
> On Wed, Oct 18, 2023 at 09:50:35PM +0300, Skyler Mäntysaari wrote:
>> On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
>>> On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
>>>> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
>>>>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>>>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>>>>>
>>>>>>>> How would I go about debugging this properly? Both sides see light,
>>>>>>>> but no link unless I'm using FreeBSD.
>>>>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
>>>>>
>>>>> Hi Skyler,
>>>>>
>>>>> Response from Intel team:
>>>>>
>>>>> In the ethtool -m output pasted I see TX and RX optical power is fine.
>>>>> Could you request fault status on both sides? Just want to check if link
>>>>> is down because we are at local-fault or link partner is at local-fault.
>>>>>
>>>>> rmmod ixgbe
>>>>> modprobe ixgbe
>>>>> ethtool -S eth0 | grep fault
>>>>>
>>>>> Since it is 10G, if our side TX is ON (power level says it is) then we
>>>>> should expect link partner RX to be locked so cannot be at Local Fault.
>>>>>
>>>>> Skyler, please gather that ethtool -S data for us.
>>>>>
>>>>> Jesse
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> Hi Jesse,
>>>>
>>>> As the other side of the link is an Juniper, I'm not quite sure how I
>>>> would gather the same data from it as it doesn't have ethtool?
>>>>
>>>> I have also somewhat given up hope on it working on VyOS and instead I
>>>> am using OPNsense for the moment but I still have VyOS installed as
>>>> well.
>>>>
>>>> Skyler
>>>
>>> Hi Jesse,
>>>
>>> I did verify that the grep doesn't yield any results on the VyOS box
>>> and all of the data returned has an value of 0. Paste of which is here:
>>> https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
>>>
>>> I really think something weird is going on with the driver in Linux as
>>> otherwise the same exact config on Juniper wouldn't work there either.
>>> The VyOS box also says that it's unable to modify autoneg settings, or
>>> speed/duplex of the interface.
>>>
>>> Skyler
>>
>> It has been  verified that the driver in kernel version 5.4.255 seems to work aka 1.3 VyOS.  Post from another user in the same thread about it: https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46
>>
>> I have also verified that the out-of-tree ixgbe driver does work, but in-kernel doesn't in kernel 6.1.58.
>>
>> Please share these findings with the correct Intel team so that this could be fixed.
>>
>> - Skyler
>>
> 
> Hi Skyler, Jesse,
> 
> I came across this very similar issue when upgrading our networking gear
> from kernel 5.15 to 6.1. Our 10G link fails with the in-tree 6.1 ixgbe
> driver but works with the out-of-tree 5.x versions. I found that my link
> issues were related to this commit:
> 
> ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.1.63&id=565736048bd5f9888990569993c6b6bfdf6dcb6d
> 
> Specifically, our 10G link works when both sides of the fiber are
> running the in-tree 6.1 ixgbe driver with this autonegotiation change.
> Our link also works when both sides are running the 5.x ixgbe drivers
> without this commit. It fails, however, when only one side has this
> commit. Our current setup compiles the in-tree 6.1 ixgbe driver with
> this commit reverted, for compatibility with our varying hardware.
> 
> I would appreciate it if anyone can cross-check my claim with their
> hardware as well. Also, would anyone be able to help explain what some
> of those registers and reg_val being written are doing?
> 
> -Ivan
> 

Hi Ivan,

It seems that for whatever reason, your reply does not appear on the lists?

- Skyler