Message-Id: <20210623075925.2610908-1-idosch@idosch.org>
Date:   Wed, 23 Jun 2021 10:59:21 +0300
From:   Ido Schimmel <idosch@...sch.org>
To:     netdev@...r.kernel.org
Cc:     davem@...emloft.net, kuba@...nel.org, jiri@...dia.com,
        andrew@...n.ch, vladyslavt@...dia.com, moshe@...dia.com,
        vadimp@...dia.com, mkubecek@...e.cz, mlxsw@...dia.com,
        Ido Schimmel <idosch@...dia.com>
Subject: [RFC PATCH net-next 0/4] ethtool: Add ability to write to transceiver module EEPROMs

From: Ido Schimmel <idosch@...dia.com>

This patchset adds write support to transceiver module EEPROMs by
extending the ethtool netlink API.

Motivation
==========

The kernel can currently dump the contents of module EEPROMs to user
space via the ethtool legacy ioctl API or the new netlink API. These
dumps can then be parsed by ethtool(8) according to the specification
that defines the memory map of the EEPROM. For example, SFF-8636 [1] for
QSFP and CMIS [2] for QSFP-DD.

In addition to read-only elements, these specifications also define
writeable elements that can be used to control the behavior of the
module. For example, controlling whether the module is put in low or
high power mode to limit its power consumption.

The CMIS specification even defines a message exchange mechanism (CDB,
Command Data Block) on top of the module's memory map. This allows the
host to send various commands to the module. For example, to update its
firmware.

Implementation
==============

The legacy ioctl API to dump module EEPROMs required drivers to parse
the contents of the EEPROM in order to understand how many bytes can be
read and dumped to user space. This meant that drivers had to be updated
to support new standards. See [3], for example.

To overcome this limitation, a new netlink-based API to dump module
EEPROMs was merged in kernel 5.13 [4]. With the new API, the kernel is
merely responsible for fetching EEPROM pages. User space then parses the
information, determines if more pages are available and instructs the
kernel to fetch them as well.

Write support for module EEPROMs employs the same approach. User space
instructs the kernel which bytes (page/offset/bank/length) to change and
to which values.

This approach allows the kernel to remain ignorant of the various
standards and avoids the need to constantly update the kernel to support
new registers / commands. More importantly, it allows advanced
functionality such as firmware update to be implemented once in user
space and shared across all the drivers that support read and write
access to module EEPROMs.

The above is achieved by adding a new command to the generic ethtool
netlink family ('ETHTOOL_MSG_MODULE_EEPROM_SET') which shares the same
attributes with the get command ('ETHTOOL_MSG_MODULE_EEPROM_GET'). See
Documentation/networking/ethtool-netlink.rst in patch #3 for detailed
description of the proposed netlink API.

Note that the new command shares the same restrictions with the existing
get command. This means, for example, that no more than 128 bytes can be
written at once and that cross-page write is forbidden. However, some
CMIS compliant modules might support "Auto Paging" which allows hosts to
"write data in large chunks, without the overhead of explicitly
programming Page changes" [2].

At this time, I cannot evaluate the benefits of "Auto Paging" as I do
not have modules that support the feature, nor a host that can write
more than 48 bytes at once. If the current restrictions prove to be a
bottleneck, they can be relaxed in the future.

ethtool(8) support
==================

The corresponding user space patches extend ethtool(8) with the ability
to change the value of a single byte in the module EEPROM. Example:

 # ethtool -M swp11 offset 0x80 page 3 bank 0 i2c 0x50 value 0x44

This is in accordance with the '-E' option which allows changing the
value of a single byte in the EEPROM of the network device.

The current command line interface is not user-friendly and also
impractical for functionality that requires many reads and writes such
as firmware update.

Therefore, the plan is to extend ethtool(8) over time with commonly
requested functionality on top of the netlink API.

Testing
=======

Tested by writing to page 3 (User EEPROM) of a QSFP-DD module:

 # ethtool -m swp11 offset 0x80 length 3 page 3 bank 0 i2c 0x50
 Offset          Values
 ------          ------
 0x0080:         00 00 00
 # ethtool -M swp11 offset 0x80 page 3 bank 0 i2c 0x50 value 0x44
 # ethtool -M swp11 offset 0x81 page 3 bank 0 i2c 0x50 value 0x41
 # ethtool -M swp11 offset 0x82 page 3 bank 0 i2c 0x50 value 0x44
 # ethtool -m swp11 offset 0x80 length 3 page 3 bank 0 i2c 0x50
 Offset          Values
 ------          ------
 0x0080:         44 41 44

Patchset overview
=================

Patches #1-#2 refactor the ethtool module EEPROM code to allow sharing
attribute validation between read and write.

Patch #3 adds the actual module EEPROM write implementation.

Patch #4 adds mlxsw support.

[1] https://members.snia.org/document/dl/26418
[2] http://www.qsfp-dd.com/wp-content/uploads/2021/05/CMIS5p0.pdf
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6af496adcbb8d4656b90a85401eeceb88d520c0d
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7dc85b599ae17fb705ffae1b7321ace4b3056aeb

Ido Schimmel (4):
  ethtool: Extract module EEPROM attributes before validation
  ethtool: Split module EEPROM attributes validation to a function
  ethtool: Add ability to write to transceiver module EEPROM
  mlxsw: core: Add support for module EEPROM write by page

 Documentation/networking/ethtool-netlink.rst  |  47 +++++
 .../net/ethernet/mellanox/mlxsw/core_env.c    |  44 ++++
 .../net/ethernet/mellanox/mlxsw/core_env.h    |   5 +
 drivers/net/ethernet/mellanox/mlxsw/minimal.c |  13 ++
 .../mellanox/mlxsw/spectrum_ethtool.c         |  14 ++
 include/linux/ethtool.h                       |  21 +-
 include/uapi/linux/ethtool_netlink.h          |   2 +
 net/ethtool/eeprom.c                          | 192 +++++++++++++++---
 net/ethtool/netlink.c                         |   7 +
 net/ethtool/netlink.h                         |   2 +
 10 files changed, 316 insertions(+), 31 deletions(-)

-- 
2.31.1

Powered by blists - more mailing lists