Message-ID: <5AABF554.3719.302C6988@Frantisek.Rysanek.post.cz>
Date:   Fri, 16 Mar 2018 17:48:20 +0100
From:   "Frantisek Rysanek" <Frantisek.Rysanek@...t.cz>
To:     netdev@...r.kernel.org
Subject: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?

Dear polite inhabitants of the "netdev" mailing list,

this is for a skunkworks project at the fringe of my job...
More of a DIY hobby thing :-) I'm tinkering and having fun.

The wizards from linux-ptp have taught me how to use the i210 for 
precise timestamping, which works fine at all copper speeds 
and likely also for gigabit fiber (SERDES) which the i210 natively 
supports (and which links fine).

The catch is: I'm trying to get this to work on 100Base-FX :-) 
which is not a native interface for the i210.
It can do SERDES only at 1Gb. But, it can do SGMII.
I've managed to find an i210-based NIC from DeLock with an SFP slot
and I've managed to find two models of 100Base-FX SGMII SFP's.

My assumption is that the SFP's have SGMII instead of SERDES
and MDIO instead of SPD I2C - and the manufacturers of the board
and of the SFP's don't dispute that. Yet they have both vaguely 
warned me in advance that it would not work work :-)
Well I've given it a try anyway, and I'm stuck at the MDIO level.

With one of the SFP's, I know for a fact that it's based on the 
BCM5461S. The actual marking on the chip goes BCM5461SA... etc.

I've managed to modify the EEPROM of the NIC with EEUPDATE.
The Device ID wouldn't change, but the "flash config words" that I 
needed were all willing to accept the change. So I can switch the 
chip to SGMII with external MDIO, and I can even see the chip
generate an MDIO read transaction right after power-up.
http://support.fccps.cz/download/adv/frr/ptp/MDIO_oscillograms.zip

And this is where I'm stuck:
the BCM chip does not respond to the i210 MDIO requests.

I've hacked the igb driver a little, adding printks where I needed 
them, to see what's going on... I believe I've also 
spotted a minor bug where some PHY detection routine
tries to read the status register before the phy.addr is even 
initialized...

This is what the Intel chip does on power up:
"read PHY reg. 0 from PHY addr <whatever I specify in the flash> ".
(= read PHY_CONTROL)


The igb driver does:

1) ret_val = hw->phy.ops.write_reg(hw, 0x1B, 0x8084);
...which succeeds, as apparently there's no "ACK bit" in the write 
sequence. The MDIO master never gets to know if the slave PHY is 
alive or not :-)

2) read "PHY control" == read PHY register 0
... this fails. The PHY just doesn't do its part, and the MDIO 
controller HW in the MAC doesn't raise the status flag saying 
"transaction completed".
The PHY just doesn't take over. It seems as if the i210 does not even 
continue sending CLK pulses - as if waiting for an active 0 (ACK?) 
from the PHY, and the ACK never comes.

I have modified the igb driver to just go ahead and try the next 
step:

3) read PHY device ID == read PHY_ID1 == PHY register 0x02.
...which fails.

I did not bother to try other registers.

The one thing I'm not certain about is the required PHY addr.
Not sure if this is hardwired in the PHY chip, or set by pin straps,
or set by soft straps in the SFP's EEPROM or something.
(Not that I've seen an EEPROM inside the SFP.)
I've noticed some earlier suggested patches by Jonathan Toppins and 
Alan Liebthal of Cumulus Networks (from mid 2015) who seem to have 
gotten the "server ATOM's" MAC to work with a BCM5461S.
Their code mentioned phy_addr = 5, which was possibly board-specific
in their hardware.

I've actually tried wrapping the steps 1) 2) 3) above in a FOR loop,
iterating over phy_addr from 1 to 31.
I can see on an oscilloscope that the i210's MDIO controller keeps 
trying different PHY addresses in the respective 5 bits of phy_addr, 
but the SFP just doesn't respond to any read transaction.
The MDC (clock) from the i210 is a nice 50:50 rectangle at 2 MHz.

I have reasons to believe that the SFP is powered. There's a P-FET in 
3.3Vcc driven by SDP3, which opens when SDP3 goes log.0. 
Which does happen - the required state of SDP3 is encoded 
in the flash and the driver also writes the CTRL_EXT at runtime 
just in case.
Also, it works in SERDES mode, where the NIC links just fine.

And, I believe I can see some erratic responses from the SFP if I 
turn off external MDIO = when the i210 SGMII tries to run with i2c 
for management access to the PHY. In i2c mode, the controller runs 
the clock line at 100 kHz (and obviously the framing/protocol is 
different) - seems to me on the scope that the PHY starts babbling 
something on the data line. The read transactions in i2c mode even 
seem to succeed, but they read garbage. Interestingly regular, 
garbage. Mii-diag says:
Basic registers of MII PHY #1:  
   7fff ff80 8000 007f 7fff ff80 8000 00ff.
And that pattern repeats always the same.

The SPF vendor has already replied that, as per spec, the BCM PHY
should support up to 2.5 MHz MDC. And, they provided a framing 
diagram which looks like a perfect IEEE clause 22.

Still I'm wondering if I should try to marry the igb driver with the 
"phylib" (which has some specific support for the BCM5461) and 
specifically with the bit-banging MDIO driver.

Bit-banging might get me around some protocol incompatibility in MDIO 
(Broadcom vs. Intel) but I would face another problem:
the way I understand it, the i210 needs to auto-configure the MAC 
block based on data it gets via MDIO autonomously from the PHY. (I 
understand that the high-speed SGMII lane for payload cannot 
transport PHY management data in-band, is that correct?)
So if I bit-bang the MDIO protocol, and even get to talk to the SFP, 
I probably won't be able to configure the MAC accordingly 
explicitly...

After a few hours of playing with this, I finally stumbled (a little 
too late) across this FAQ entry in an Intel i210/i211 FAQ PDF:

"For external PHY use SerDes SKU and configure to use SGMII with 
SGMII capable PHY such as a Marvell 88e1112.  Be advised, ND has not 
been able to inter-operate with BRCM SGMII PHY so far."

Which does not explain any details. The full PDF is here:
https://www.intel.com/content/dam/www/public/us/en/documents/faqs/ethe
rnet-controller-i210-i211-faq.pdf

Anyway... if you've read this far, thanks for your kind attention
- and, any sort of comments are welcome :-)

Frank Rysanek

Powered by blists - more mailing lists