netdev - Re: ethtool fails to read some QSFP+ modules.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <10ec2a87-5d9e-40a7-a4ea-1e50fd8109e6@dan.merillat.org>
Date: Tue, 2 Jul 2024 15:30:58 -0400
From: Dan Merillat <git@....merillat.org>
To: netdev <netdev@...r.kernel.org>
Cc: Dan Merillat <git@....merillat.org>, Ido Schimmel <idosch@...sch.org>
Subject: Re: ethtool fails to read some QSFP+ modules.

Sorry, I'm not subscribed to netdev and didn't get a CC so I'm trying to follow via the web archives
and I can't see who was CC'd.

On Mon, Jul 01, 2024 at 10:28:39AM +0300, Ido Schimmel wrote:
> On Sun, Jun 30, 2024 at 01:27:07PM -0400, Dan Merillat wrote:
> > 
> > I was testing an older Kaiam XQX2502 40G-LR4 and ethtool -m failed with netlink error.  It's treating a failure to read
> > the optional page3 data as a hard failure.
> > 
> > This patch allows ethtool to read qsfp modules that don't implement the voltage/temperature alarm data.
> 
> Thanks for the report and the patch. Krzysztof OlÄ™dzki reported the same
> issue earlier this year:
> https://lore.kernel.org/netdev/9e757616-0396-4573-9ea9-3cb5ef5c901a@ans.pl/
> 
> Krzysztof, are you going to submit the ethtool and mlx4 patches?
> 
> > From 3144fbfc08fbfb90ecda4848fc9356bde8933d4a Mon Sep 17 00:00:00 2001
> > From: Dan Merillat <git@....eginity.com>
> > Date: Sun, 30 Jun 2024 13:11:51 -0400
> > Subject: [PATCH] Some qsfp modules do not support page 3
> > 
> > Tested on an older Kaiam XQX2502 40G-LR4 module.
> > ethtool -m aborts with netlink error due to page 3
> > not existing on the module. Ignore the error and
> > leave map->page_03h NULL.
> 
> User space only tries to read this page because the module advertised it
> as supported. It is more likely that the NIC driver does not return all
> the pages. Which driver is it?

Same as Kyrzysztof, a connectx3 board.  I'm using mlx4_core/en in the stock 6.9.3 kernel.

Raw dump:
# ethtool -m enp65s0d1 raw on | hexdump -C 
00000000  0d 05 00 0f 00 00 00 00  00 44 44 00 00 44 44 00  |.........DD..DD.|
00000010  00 00 00 00 00 00 30 a4  00 00 82 3b 00 00 00 00  |......0....;....|
00000020  00 00 00 10 00 11 00 31  00 00 36 ff 36 ff 36 ff  |.......1..6.6.6.|
00000030  36 83 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |6...............|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 03 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  0d 80 07 02 00 00 00 00  00 00 00 00 67 00 02 00  |............g...|
00000090  00 00 00 40 4b 41 49 41  4d 20 43 4f 52 50 20 20  |...@...AM CORP  |
000000a0  20 20 20 20 00 14 ed e4  58 51 58 32 35 30 32 20  |    ....XQX2502 |
000000b0  20 20 20 20 20 20 20 20  31 41 66 58 05 14 46 14  |        1AfX..F.|
000000c0  00 00 00 92 4b 44 37 30  32 30 31 31 35 39 20 20  |....KD70201159  |
000000d0  20 20 20 20 31 37 30 32  30 31 30 30 08 00 00 0d  |    17020100....|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100

The ioctl path handles it properly when ethtool is built with --disable-netlink, since it checks the
total buffer size returned by the kernel and does not set map->page_03h if the size does not match.

Status byte 2, bit 2 is 'Upper memory flat or paged.   0x1b: Flat memory.  0x0b: Paging (at least upper
page 0x03h supported)'

SFF8636_STATUS_FLAT_MEMORY would be clearer, as page_00h[STATUS_2] & PAGE_3_PRESENT is really a
negative test.

I confirmed that status 2 is 0x00, which would indicate the module supports paged memory.

So either this cheap module is non-compliant or there's a problem in the driver/firmware for
connectx3/3pro cards.  Either way, it would be better to warn about the non-compliance and still
display  the optical information gathered rather than a fatal error that shows nothing.  If it can
also be fixed in the mlx4_core driver to properly read page 3, that would be better, but that
will require someone with more knowledge of mellanox/nVidia internals than I have.

If anyone wants to test with a different adapter, these xqx2502 modules are dirt cheap on ebay, only $4 when 
I bought mine a few weeks ago.