| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20180926213436.GA4116@hmswarspite.think-freely.org>
Date: Wed, 26 Sep 2018 17:34:36 -0400
From: Neil Horman <nhorman@...driver.com>
To: Chris Preimesberger <chrisp@...nsition.com>
Cc: "linville@...driver.com" <linville@...driver.com>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: bug: 'ethtool -m' reports spurious alarm & warning threshold
values for QSFP28 transceivers
On Wed, Sep 26, 2018 at 07:29:23PM +0000, Chris Preimesberger wrote:
> Hello,
>
> I'm re-sending in plain text per the auto-reply from a spam filter. I have attached some text files this time, which explain the situation below, in case the below email's font & formatting is now too messed up for easy comprehension.
>
> Thank you and best regards.
>
>
> Chris Preimesberger | Test & Validation Engineer
> Transition Networks, Inc.
>
> chrisp@...nsition.com
> direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
> ________________________________________
>
>
>
This is just a drive by guess, but I think this is a driver issue.
Issue 1 seems like a red herring, cat doesn't modify output, nor does ethtool
know if its output is going to a console or a pipe, its all the same. And given
issue 2 (that the output of the thresholds, etc are spurriously changing and
wrong), suggests that they are spurriously changing and wrong regardless of what
cat does.
That said, I think issue two is a problem with the mlx4 driver. Specifically
that the driver is copying garbage data.
The three ethtool functions at work here are:
mlx4_en_get_module_info
mlx4_en_get_module_eeprom
mlx4_get_module_info
When you run ethtool -m on this driver, the kernel calls mlx4_en_get_module_info
to determine the length of the eeprom, and that value will be either 256 or 512
bytes. Lets assume that the value is 256 for the sake of argument
Next it calls mlx4_en_get_module_eeprom, passing in that size 256 to actually
read the eeprom data, which in turn calls mlx4_get_module_info to fetch the data
from hardware, again, passing in 256 as the size for the first call (theres a
loop, but it will only get executed once in this scenario)
mlx4_get_module_info then issues the appropriate mailbox commands to dump the
eeprom. Here it starts to go sideways. The mailbox buffer allocated for the
return data is of type mlx4_mad_ifc, which has some front matter information and
a data buffer that is 192 bytes long!
A little further down in the function, size gets restricted if the buffer
crosses a page boundary, but given that the size is 256 on the first call here,
and offset is zero on the first call, we're not crossing anything, so size
remains unchanged.
The output mailbox buffer outmad->data (a 192 byte array), then gets cast to a
sturt mlx4_cable_info structure, which has its own internal data buffer that is
only 48 bytes long.
The memcpy in this functionthen copies cable_info->data to the buffer that gets
returned to ethtool, but it copies size bytes (256), even though the source data
buffer is only 48 bytes long. That 48 byte array is embedded in the larger 192
byte structure, so there won't be a panic on the overrun, but theres no telling
what garbage is in the buffer beyond those first 48 bytes. Even if the
remaining 144 bytes have valid eeprom data, its less than the required 256
bytes. The additional copy may cause a panic, but if the buffer commonly bumps
up against other allocated memory, that will go unnoticed.
after the memcpy, mlx4_get_module_info just returns the size of the passed in
buffer (256), and so the calling function thinks its work is done, and lets the
kernel send back the buffer with garbage data to ethtool.
I think the mlx4 guys have some work to do here.
My $0.02
Neil
>
>
> From: Chris Preimesberger
> Sent: Wednesday, September 26, 2018 2:14 PM
> To: 'linville@...driver.com'; 'netdev@...r.kernel.org'
> Subject: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
>
> Hello John, All,
>
>
> I think I may have found a bug or two in ethtool, with respect to its reporting of a QSFP28 transceiver's diagnostic information. Ethtool seems to correctly report all diagnostic information about QSFP28 transceivers, except for the transceiver's warning and alarm thresholds. I'm not sure whether the spurious warning and alarm values that get reported are the fault of ethtool or my NIC/driver, and I have no other models of 100GbE NICs to test with. I've contacted Mellanox support about this, and they point the finger at ethtool. Can these issues be investigated by ethtool developers? Here is some background information about the equipment and software used when I observe these issues:
>
> Equipment used:
> NIC: Mellanox ConnectX-4 100GbE, part number MCX415A-CCAT
> Transceiver: Any 40Gb or 100Gb QSFP28 transceiver installed in the NIC (Intel, Mellanox, Transition Networks, etc..)
>
> Software used:
> Ubuntu 18.04 with the distro's packaged NIC driver and ethtool v4.15
> also tested were ethtool v4.18 compiled from source and the current Mellanox OFED driver.
>
> All test scenarios produced the same bugs.
>
>
> Bug #1. Ethtool's reporting of the installed transceiver's alarm and warning thresholds will differ, depending on whether or not ethtool is piped to another command. Example commands are below, with their respective differing output values highlighted:
>
>
> tech1@D8:~$ sudo ethtool -m enp1s0
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3241 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9048 mW / -0.43 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7014 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 0.000 mA
> Laser bias current low alarm threshold : 0.000 mA
> Laser bias current high warning threshold : 0.000 mA
> Laser bias current low warning threshold : 0.000 mA
> Laser output power high alarm threshold : 0.0000 mW / -inf dBm
> Laser output power low alarm threshold : 0.0000 mW / -inf dBm
> Laser output power high warning threshold : 0.0000 mW / -inf dBm
> Laser output power low warning threshold : 0.0000 mW / -inf dBm
> Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
> Module voltage high alarm threshold : 0.0000 V
> Module voltage low alarm threshold : 0.0000 V
> Module voltage high warning threshold : 0.0000 V
> Module voltage low warning threshold : 0.0000 V
> Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power high warning threshold : 0.0000 mW / -inf dBm
> Laser rx power low warning threshold : 0.0000 mW / -inf dBm
>
>
> tech1@D8:~$ sudo ethtool -m enp1s0 | cat
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3249 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9043 mW / -0.44 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 16.448 mA
> Laser bias current low alarm threshold : 16.448 mA
> Laser bias current high warning threshold : 16.448 mA
> Laser bias current low warning threshold : 16.448 mA
> Laser output power high alarm threshold : 0.8224 mW / -0.85 dBm
> Laser output power low alarm threshold : 0.8250 mW / -0.84 dBm
> Laser output power high warning threshold : 0.8264 mW / -0.83 dBm
> Laser output power low warning threshold : 2.6983 mW / 4.31 dBm
> Module temperature high alarm threshold : 110.12 degrees C / 230.22 degrees F
> Module temperature low alarm threshold : 84.34 degrees C / 183.82 degrees F
> Module temperature high warning threshold : 44.12 degrees C / 111.42 degrees F
> Module temperature low warning threshold : 67.27 degrees C / 153.08 degrees F
> Module voltage high alarm threshold : 2.9728 V
> Module voltage low alarm threshold : 2.6990 V
> Module voltage high warning threshold : 0.8274 V
> Module voltage low warning threshold : 2.2538 V
> Laser rx power high alarm threshold : 2.5458 mW / 4.06 dBm
> Laser rx power low alarm threshold : 2.6992 mW / 4.31 dBm
> Laser rx power high warning threshold : 2.9801 mW / 4.74 dBm
> Laser rx power low warning threshold : 2.8526 mW / 4.55 dBm
>
>
> Bug # 2. All of the alarm and warning threshold values reported in the above commands are spurious.
> At first glance, one would assume that the threshold values reported by the piped ethtool command are correct, but they're not. I know the programmed values for the above transceiver, so that makes it easy for me to spot the spurious values, but even without knowing the programmed values of a given transceiver, one can use logic to detect when the ethtool displayed values don't make sense.
> For example, lets scrutinize the values for voltage warnings and alarms reported by ethtool on this transceiver. We will look at each voltage threshold, and scrutinize that value relative to the other voltage thresholds, and look for contradictions to determine whether the reported values seem legit.
> Known ethtool
> Actual Reported
> Values Values
> High Voltage Alarm 3.70V 2.9728 V
> High Voltage Warning 3.59V 0.8274 V
> (Operating spec = 3.30V)
> Low Voltage Warning 3.00V 2.2538 V
> Low Voltage Alarm 2.90V 2.6990 V
>
> Contradictions for the ethtool reported voltage warning and alarm thresholds:
> 1. The high voltage alarm should occur at higher voltage than the operating voltage, but ethtool didn't report that.
> 2. The high voltage warning should occur at higher voltage than the low voltage warning and alarm, but ethtool didn't report that.
> 3. The low voltage warning should occur at higher voltage than the low voltage alarm, but ethtool didn't report that.
> 4. The low voltage alarm should occur at a lower voltage than any of the other voltage warnings and alarms, but ethtool didn't report that.
> 5. The current voltage value was reported as 3.3249V, which should trigger high voltage warning and alarm, according to the reported thresholds, but no warnings or alarms are indicated.
>
> Each of the 4 voltage thresholds reported by ethtool have contradictions, so we know something is not right. This same kind of logic can be applied to the thresholds for temperature, laser TX power, etc.. to find that those values are also spurious.
>
>
> Installing the above transceiver in a Cisco switch reveals that the Cisco correctly retrieves the true warning and alarm threshold values from the transceiver's EEPROM, so we trust that the transceiver has been correctly programmed. Cisco CLI output for that transceiver shown here:
>
> switch# show interface ethernet 1/3 transceiver details
> Ethernet1/3
> transceiver is present
> type is QSFP-100G-CWDM4-MSA-FEC
> name is TRANSITION
> part number is TNQSFP100GCWDM4
> revision is 1A
> serial number is TN02000302
> nominal bitrate is 25500 MBit/sec per channel
> Link length supported for 9/125um fiber is 2 km
> cisco id is 17
> cisco extended id number is 252
>
> Lane Number:1 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.44 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:2 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.20 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:3 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.21 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.96 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:4 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.72 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.59 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> switch#
>
>
> Any help with these issues is greatly appreciated. If you have any questions or advice, please let me know. I'll be glad to continue troubleshooting this until it's resolved. Thank you.
>
>
> Chris Preimesberger | Test & Validation Engineer
> Transition Networks, Inc.
>
> chrisp@...nsition.com
> direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
> ________________________________________
>
>
>
>
>
>
>
> For comparison to ethtool's output that shows incorrect threshold values, when installing the same transceiver in a Cisco Nexus switch, and issuing the Cisco command "show interface ethernet 1/3 transceiver details", the switch correctly correctly reads/displays the transceiver's Alarm and Warning thresholds, as shown below:
>
>
> switch# show interface ethernet 1/3 transceiver details
> Ethernet1/3
> transceiver is present
> type is QSFP-100G-CWDM4-MSA-FEC
> name is TRANSITION
> part number is TNQSFP100GCWDM4
> revision is 1A
> serial number is TN02000302
> nominal bitrate is 25500 MBit/sec per channel
> Link length supported for 9/125um fiber is 2 km
> cisco id is 17
> cisco extended id number is 252
>
> Lane Number:1 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.44 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:2 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.20 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:3 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.21 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.96 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:4 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.72 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.59 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> switch#
>
>
> Look at each line in the ethtool output below that includes the word "threshold". This file has been hand-edited to show the threshold values that have been programmed into the transceiver, which should be displayed by ethtool. The threshold values shown below are copied and pasted from the output of the Cisco NX-OS command "show interface ethernet 1/3 transceiver details", while the transceiver was installed in a Cisco Nexus switch.
>
> Note - I only copied the threshold values in the units that were displayed by the Cisco switch. The "?" symbols are just a placeholder for the converted values; I was too lazy to do conversions between dBm and mW, or between degrees C and degrees F. Ethtool would be expected to report the true / converted values.
>
>
>
>
> tech1@D8:~$ sudo ethtool -m enp1s0
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3233 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9052 mW / -0.43 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6948 mW / -1.58 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 75.000 mA
> Laser bias current low alarm threshold : 10.000 mA
> Laser bias current high warning threshold : 70.000 mA
> Laser bias current low warning threshold : 15.000 mA
> Laser output power high alarm threshold : ? mW / 4.49 dBm
> Laser output power low alarm threshold : ? mW / -8.50 dBm
> Laser output power high warning threshold : ? mW / 3.49 dBm
> Laser output power low warning threshold : ? mW / -7.52 dBm
> Module temperature high alarm threshold : 80.00 degrees C / ? degrees F
> Module temperature low alarm threshold : -10.00 degrees C / ? degrees F
> Module temperature high warning threshold : 75.00 degrees C / ? degrees F
> Module temperature low warning threshold : -5.00 degrees C / ? degrees F
> Module voltage high alarm threshold : 3.7000 V
> Module voltage low alarm threshold : 2.9000 V
> Module voltage high warning threshold : 3.5900 V
> Module voltage low warning threshold : 3.0000 V
> Laser rx power high alarm threshold : ? mW / 4.49 dBm
> Laser rx power low alarm threshold : ? mW / -14.55 dBm
> Laser rx power high warning threshold : ? mW / 3.49 dBm
> Laser rx power low warning threshold : ? mW / -12.51 dBm
>
>
> Look at each line in the ethtool output below that includes the word "threshold". This file shows the actual output from ethtool v4.18, when the output is not piped to another command. Notice that all of the displayed threshold values are 0 (which is incorrect), while other values report as expected.
>
> tech1@D8:~$ sudo ethtool -m enp1s0
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3241 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9048 mW / -0.43 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7014 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 0.000 mA
> Laser bias current low alarm threshold : 0.000 mA
> Laser bias current high warning threshold : 0.000 mA
> Laser bias current low warning threshold : 0.000 mA
> Laser output power high alarm threshold : 0.0000 mW / -inf dBm
> Laser output power low alarm threshold : 0.0000 mW / -inf dBm
> Laser output power high warning threshold : 0.0000 mW / -inf dBm
> Laser output power low warning threshold : 0.0000 mW / -inf dBm
> Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
> Module voltage high alarm threshold : 0.0000 V
> Module voltage low alarm threshold : 0.0000 V
> Module voltage high warning threshold : 0.0000 V
> Module voltage low warning threshold : 0.0000 V
> Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power high warning threshold : 0.0000 mW / -inf dBm
> Laser rx power low warning threshold : 0.0000 mW / -inf dBm
>
>
>
> Look at each line in the ethtool output below that includes the word "threshold". This file shows the actual output from ethtool v4.18, when the ethtool output is piped to another command. Notice that all of the displayed threshold values are spurious while other values report as expected.
>
> tech1@D8:~$ sudo ethtool -m enp1s0 | cat
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3249 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9043 mW / -0.44 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 16.448 mA
> Laser bias current low alarm threshold : 16.448 mA
> Laser bias current high warning threshold : 16.448 mA
> Laser bias current low warning threshold : 16.448 mA
> Laser output power high alarm threshold : 0.8224 mW / -0.85 dBm
> Laser output power low alarm threshold : 0.8250 mW / -0.84 dBm
> Laser output power high warning threshold : 0.8264 mW / -0.83 dBm
> Laser output power low warning threshold : 2.6983 mW / 4.31 dBm
> Module temperature high alarm threshold : 110.12 degrees C / 230.22 degrees F
> Module temperature low alarm threshold : 84.34 degrees C / 183.82 degrees F
> Module temperature high warning threshold : 44.12 degrees C / 111.42 degrees F
> Module temperature low warning threshold : 67.27 degrees C / 153.08 degrees F
> Module voltage high alarm threshold : 2.9728 V
> Module voltage low alarm threshold : 2.6990 V
> Module voltage high warning threshold : 0.8274 V
> Module voltage low warning threshold : 2.2538 V
> Laser rx power high alarm threshold : 2.5458 mW / 4.06 dBm
> Laser rx power low alarm threshold : 2.6992 mW / 4.31 dBm
> Laser rx power high warning threshold : 2.9801 mW / 4.74 dBm
> Laser rx power low warning threshold : 2.8526 mW / 4.55 dBm
> tech1@D8:~$
>
Powered by blists - more mailing lists