Message-ID: <ZADcSwvmwt8jYxWD@shell.armlinux.org.uk>
Date:   Thu, 2 Mar 2023 17:26:35 +0000
From:   "Russell King (Oracle)" <linux@...linux.org.uk>
To:     Jakub Kicinski <kuba@...nel.org>
Cc:     Richard Cochran <richardcochran@...il.com>,
        Köry Maincent <kory.maincent@...tlin.com>,
        andrew@...n.ch, davem@...emloft.net, f.fainelli@...il.com,
        hkallweit1@...il.com, netdev@...r.kernel.org,
        Thomas Petazzoni <thomas.petazzoni@...tlin.com>,
        Maxime Chevallier <maxime.chevallier@...tlin.com>
Subject: Re: [PATCH RFC net-next] net: phy: add Marvell PHY PTP support
 [multicast/DSA issues]

On Thu, Mar 02, 2023 at 08:49:32AM -0800, Jakub Kicinski wrote:
> On Thu, 2 Mar 2023 11:49:26 +0000 Russell King (Oracle) wrote:
> > (In essence, because of all the noise when trying the Marvell PHY with
> > ptp4l, I came to the conlusion that NTP was a far better solution to
> > time synchronisation between machines than PTP would ever be due to
> > the nose induced by MDIO access. However, I should also state that I
> > basically gave up with PTP in the end because hardware support is
> > overall poor, and NTP just works - and I'd still have to run NTP for
> > the machines that have no PTP capabilities. PTP probably only makes
> > sense if one has a nice expensive grand master PTP clock on ones
> > network, and all the machines one wants to synchronise have decent
> > PTP implementations.)
> 
> Don't wanna waste too much of your time with the questions since
> I haven't done much research but - wouldn't MAC timestamp be a better
> choice more often (as long as it's a real, to-spec PTP stamp)? 
> Are we picking PHY for historical reasons?
> 
> Not that flipping the default would address the problem of regressing
> some setups..

There is the argument in PTP that having the PHY do the timestamping
results in the timestamps being "closer to the wire" meaning that
they will not be subject to any delays between the packet leaving
the MAC and being transmitted on the wire. Some PHYs have FIFOs or
other buffering which introduces a delay which PTP doesn't desire.

TI has a blog on this:

https://e2e.ti.com/blogs_/b/analogwire/posts/how-to-implement-ieee-1588-time-stamping-in-an-ethernet-transceiver

However, what is failed to be mentioned there is that yes, doing
PTP at the PHY means one can accurately trigger a capture of the
timestamp from the PHC when the SFD is detected on the media. That
is a great advantage, but is really only half the story.

If the PHC (hardware counter) in the PHY can't be accurately
synchronised, for example, it has a 1.5ppm second-to-second
variability, then the resulting timestamps will also have that
same variability. Another way to look at it is they will appear to
have 1.5ppm of noise. If this noise is random, then it becomes
difficult to filter that noise out, and results in jitter.

However, timestamping at the MAC may have only 40ppb of variability,
but have a resulting delay through the PHY. The delay through the
PHY will likely be constant, so the timestamps gathered at the MAC
will have a constant error but have much less noise.

Things change if one can use hardware signals to synchronise the
PHC, because then we become less reliant on a variable latency
accessing the PHY over the MDIO bus. The hardware event capture
allows the PHC to be captured on a hardware signal, software can
then read that timestamp, and if the hardware event is a PPS,
then that can be used to ensure that the PHC is ticking at the
correct rate. If the PPS is also aligned to a second boundary,
then the hardware PHC can also be aligned. With both, the latency
of the MDIO bus becomes irrelevant, and PTP at the PHY becomes a
far more preferable option.

Note that things which can influence the latency over the MDIO bus
include how many PHYs or other devices are also on it, and the
rate at which accesses to those PHYs are performed. Then there's
latency from kernel locking and scheduling. Maybe interrupt
latency if the MDIO bus driver uses interrupts.

When I created the generic Marvell PHY PTP layer (my patch set
for Marvell PHYs) I tried very hard to eliminate as many of these
sources of variable latency as possible - such as avoiding taking
and releasing locks on every MDIO bus access. Even with that, I
could not get the PHY's PHC to synchronise any better than 1.5ppm,
vs 40ppb for the PP2 MAC's PHC.

The last bit of consideration is whether the PHCs can be synchronised
in hardware. If one has multiple ethernet interfaces, no hardware
synchronisation of PHY PHCs, but also have MAC PTP support which is
shared across the ethernet interfaces, why would one want to use the
PHY with software based synchronisation of their individual PHCs.

So, to wrap this email up... if one has hardware purposely designed
for PTP support, which uses the PTP hardware signals from the
various PHCs, then one would want to use the PHY based timestamping.
If one doesn't have that, then one would want to use something that
gives the best timestamps, which may not necessarily be the PHY.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Powered by blists - more mailing lists