Date: Sun, 7 May 2023 16:32:25 +0000
From: "Stern, Avraham" <avraham.stern@...el.com>
To: Richard Cochran <richardcochran@...il.com>
CC: "Greenman, Gregory" <gregory.greenman@...el.com>, "kuba@...nel.org"
	<kuba@...nel.org>, "linux-wireless@...r.kernel.org"
	<linux-wireless@...r.kernel.org>, "johannes@...solutions.net"
	<johannes@...solutions.net>, "netdev@...r.kernel.org"
	<netdev@...r.kernel.org>
Subject: RE: pull-request: wireless-next-2023-03-30

Hi Richard,

I will try to explain my point of view on the topic.
I tried to highlight the areas where wifi may be different from ethernet.
The below is only my perspective, in order to get everyone on the same page.
Please comment/correct me wherever you find fit as your perspective is definitely broader than mine.


The PTP protocol consists of 2 primary parts:
 a. best master selection algorithm using "general PTP messages" (messages that are not timestamped)
 b. time sync, using "event PTP messages" (messages that are timestamped)

A. Master selection algorithm
----------------------------------------
Regarding the master selection, PTP requires that announce messages are delivered to all PTP ports within a ptp communication path (which may include switches, bridges etc.).
For that purpose, the 1588-2019 standard defines that "Announce messages are either sent in multicast or the Announce information is replicated to all PTP Ports in the PTP
Communication Path using unicast PTP messages." (section 6.2). It seems that multicast frames are used to allow the announce messages to propagate through transparent clocks or
components of the network that does not support PTP, but has not effect when the receiving PTP port is a boundary clock or an ordinary clock since the standard specifies that
"All PTP messages except PTP management messages are terminated at a Boundary Clock" (and ordinary clocks only have 1 port).
In wifi networks, the usage of multicast frames is limited since only the AP can send multicast to all associated stations. A station that wants to send a multicast frame sends it as
a unicast frame to the AP and the AP then re-sends it as multicast to all other stations.
However, the 1588-2019 standard specifies that "Special Ports shall be used to transfer time over a network where the time transfer is
not based on the use of PTP timing messages" (which is the case for wifi networks. I will elaborate more on that in the time sync section) and that
"Special Ports shall be used only on Ordinary Clocks or Boundary Clocks" (section 11.5.1), so on wifi networks there will be only ordinary and boundary clocks, which means using
unicast instead of multicast should work just the same. Since the stations must be associated to the AP (otherwise they cannot exchange data frames at all, so can't send/receive the
announce messages to begin with), the unicast addresses are known both at the station and the AP side.
So to summarize, using the unicast option for announce messages should work on wifi networks without any changes required.
Let's examine some basic topologies:
1. The basic wifi network consists of an AP and several stations:
     Station A <---------   AP --------> station B
     
     In the common case, the AP will be master and stations A and B will follow.

2. In a Case like:
     GPS Radio ------> Station-A ~~~~~~ AP ~~~~~~ Station-B
           1PPS              WiFi      WiFi

    The AP will act as a boundary clock, with the port connected to station A as a slave, and the port connected to station B as master.

In addition, the AP can also bridge Ethernet and wifi networks, acting as the boundary clock, e.g. as a slave on the ethernet port and master on the wifi port.


B. Time synchronization
--------------------------------
     PTP defines two mechanisms for time synchronization which are used over ethernet: The delay request-response mechanism and The peer-to-peer delay mechanism.
     Both use a similar mechanism to measure the path (or link) delay:
     1. transmit a data frame from peer A to peer B, save the transmission time (t1) and reception time (t2).
     2. transmit a data frame from peer B to peer A, save the transmission time (t3) and reception time (t4).
     3. send (t1, t4) or (t2, t3) to the other peer.

     Using these mechanisms as is over a wifi link is subject to the following drawbacks:
	(i)      The 1588-2019 standard outlines that these mechanisms "assume that the master-to-slave and slave-to-master propagation times are equal.
                        Any asymmetry in propagation time introduces an error in the computed value of the clock offset. The computed mean propagation time differs
                        from the actual propagation times due to the asymmetry."  (section 6.6.3).
                        However, in wifi networks the station may be moving, which makes the propagation delay change over time due to the changing distance.
                        Because of wifi power save mechanisms, the time between two data frames may be relatively large (e.g. an AP with a typical beacon interval of 100 TUs
                        and DTIM of 3, the station may be in power save for more than 300TUs, so the time between the sync message and the delay request message may get close to
                        even 500ms). So when using data frames the propagation delay is likely to be asymmetric, which will lead to time sync error.
             (ii)      over wifi, a frame may be retransmitted (if not acked by the receiving station). This may lead to a situation where station A takes the timestamp t1 on the
                        first transmission while station B takes timestamp t2 on one of the retransmissions (or vice versa). This will introduce a significant error since t1 and t2 are not
                        referring to the same event.


    The 802.11 standard defines mechanisms for measuring the path delay, called Timing Measurement (TM) and Fine Timing Measurement (FTM).
    These mechanisms use a similar principle for calculating the path delay as the PTP mechanisms, but the timestamps are recorded on a management frame and the corresponding ack.
    This eliminates issue (i) , since the ack must be transmitted 16us after the frame is received (802.11 definitions). This make the path delay symmetric for any practical use-case.
    Using the ack for measuring the path delay also implies better usage of the network: while in the delay request-response mechanism, 4 frames are potentially needed for 1 measurement,
    in FTM 3 frames yield 2 measurement results (which can be averaged to get better accuracy, for example).
    In addition, the 802.11 standard defines the retransmissions behavior for these specific frame types to eliminate issue (ii).

    In order to use the pre-defined 802.11 mechanisms to measure the path delay over wifi, the 1588-2019 defined a new port type - the special port.
    (as stated in 11.5.2.2.2: "This interface is new to this edition and is introduced to facilitate the use within the PTP protocol of media
     that implement their own methodology for measuring path delay and transferring time, for example, IEEE Std 802.11").

    A special port has a media-dependent and media-independent parts.
    The media-independent part is responsible for managing the master slave states and calculating the time synchronization using the inputs from the media dependent layer.
    The media-dependent layer measures mean path delay and sends and receives time synchronization information (passed to the media-independent part).
    The 1588-2019 defines the interface between the two parts, called MDMI interface.
    The operation of the media-dependent part for 802.11 is further defined in the 802.1AS-2020 standard, which defines also the state-machines for using TM or FTM
    to measure the path delay, the measurement parameters etc.

    To summarize, the standards (1588 and 802.1AS) mandate the usage of special ports and TM/FTM for time sync over wifi, for the former mentioned reasons (and probably more).

    The 1588-2019 also defines the usage of a special port for boundary clock and ordinary clock only (but not transparent clock). This is a results of using a medium specific method to measure
    the path delay, which is defined point to point, which is not suitable for a transparent clock.
    This makes the "AP as transparent clock" option not supported by the standard (the AP can be a boundary clock in the station -> AP -> station topology).
    (theoretically, the AP could be used as transparent clock if unicast data frames where used for timestamping, but it would be subject to the problems mentioned).

There are also implementation considerations to favor the usage of TM/FTM over timestamping data frames:
Recording an accurate Rx timestamp on a wifi packet is difficult because the received signal may have interference, reflections etc.
Multipath detection algorithms are usually used to calculate a more accurate timestamps. There algorithms are relatively heavy for network cards. Having to calculate them
for each data packet will introduce an unnecessary overhead.  Of course, this can be minimized by calculating the Rx timestamp only on data packets that are used for time sync,
but network cards usually don't inspect data packets for their content, so they may not be able to easily tell which data packets should be timestamped.
On the other hand, many network cards already support FTM (for ranging) so they already have the ability to timestamp FTM frames.

After deciding to follow the standards, the next question is which protocol should be preferred, TM or FTM?
As implied by its name, FTM has better accuracy. While TM has units of 10ns, FTM has units of picoseconds.
And as formerly mentioned, many wifi vendors already support FTM for ranging (which TM is of course not suitable for because of the low accuracy).
So it seems FTM should be preferred if possible. (in fact, TM is probably only left for backward compatibility, but  I don't know anyone who already uses it for time sync?)
Anyway, adding support for both is also an option, since the building blocks are the same and the TM protocol is fairly simple.


Kernel interfaces
------------------------
The kernel currently have 2 interfaces for the PTP time sync.
The first one is the PHC interface for devices to expose clock operations.
This interface is related only to the media-independent part of the port, and thus can be used as-is for wifi interfaces too.

The second interface is the SO_TIMESTAMPING socket options, for timestamping data frames sent/received over a socket.
This kernel API is not suitable for wifi which will use TM/FTM, since TM/FTM uses 802.11 management frames for time sync,
not data frames. 802.11 management frames are not sent over a regular socket but using NL80211 APIs.
In addition, for TM/FTM the timestamp of the ack is also needed, which is not supported by the SO_TIMESTAMPING socket options.
Thus another interface is needed for TM/FTM timestamping.

The kernel already has NL80211 APIs for triggering FTM measurements. However, these APIs are tailored for measuring the range
from the AP, and the result reports only the distance from the AP, but not the timestamps collected for the FTM frames. As such,
it is not useful as is for time sync because for time sync the timestamps are needed for the media-independent part to calculate the sync.
In addition, as specified in 802.1AS-2020, for time sync the FTM frames need to include a vendor IE with all the PTP data (sync or followup messages).
The current APIs do not allow adding IEs to the frames used in the measurement.
Extending these APIs to support time sync also seems wrong since the measurement API triggers a burst (i.e. many FTM frames, each one produces a measurement results,
and the driver reports one accumulated result according to it's own logic - e.g. average) while for time sync each FTM frame should have a different PTP data, so this will
make the API complicated and will require massive changes in drivers to support.

Another option is to add to NL80211 the capability to timestamp TM/FTM frames. This is equivalent to the SO_TIMESTAMPING socket options in a sense - telling the driver to
timestamp the required frames only and report the timestamps to userspace.
This is the interface we introduced in our change.

This API change requires small effort from drivers/NICs - timestamp TM/FTM frames only, per request. Since many hw vendors already support FTM which means they have the
ability to timestamp these frames, supporting the new API comes down to reporting these timestamps to the driver. It is likely that any vendor that supports FTM will be able to
support the new API with minimal effort.


So the basic idea was to provide userspace similar APIs to what's used over ethernet - the PHC clock and a way to get timestamps for the desired frames (data over ethernet, TM/FTM over wifi).
>From that point on, the userspace implementation, which already has the media-independent part, will just need to add the wifi media dependent part (i.e. the TM/FTM state machine define in 802.1AS)
and pass the reported timestamps to the media-independent part.
Then time-sync will be supported by all vendors that implement the timestamping and the PHC clock  APIs.
(but that is of course just a suggestion and will require more in-depth design. Just added it here to describe how I imagined the big picture).


>> IMO, the simplest way that will unlock many use cases is to provide time stamps for single unicast frames, like in ieee80211_rx_status.device_timestamp and expose an adjustable PHC using timecounter/cyclecounter over the free running usec clock.  Then you could synchronize client/AP over unicast IPv4 PTP >> (for example) with no user space changes needed, AND it would work on all radios, even those that don't implement FTM.

I Agree, this would work without any userspace changes, but it is not with accordance to the standards (which have their considerations , which I tried to review some of above).
And it still requires devices to timestamp data frames. The suggested API also doesn't require devices to implement FTM, only to timestamp FTM frames.

I hope this makes things a little bit more clear.
Please let me know what you think.

Thanks,
Avi.





     



-----Original Message-----
From: Richard Cochran <richardcochran@...il.com> 
Sent: Wednesday, April 26, 2023 05:44
To: Stern, Avraham <avraham.stern@...el.com>
Cc: Greenman, Gregory <gregory.greenman@...el.com>; kuba@...nel.org; linux-wireless@...r.kernel.org; johannes@...solutions.net; netdev@...r.kernel.org
Subject: Re: pull-request: wireless-next-2023-03-30

On Tue, Apr 25, 2023 at 07:03:50AM +0000, Stern, Avraham wrote:

> Having the timestamps of the frames seemed like a basic capability that userspace will need to implement ptp over wifi, regardless of the selected approach.

Having time stamps on unicast PTP frames would be a great solution.
But I'm guessing that you aren't talking about that?

> Apparently you had other ways in mind, so I would love to have that discussion and hear about it.  

Let's back up a bit.  Since you would like to implement PTP over Wifi in Linux, may I suggest that the first step is to write up and publish your design idea so that everyone gets on the same page?

Your design might touch upon a number of points...

- Background
  - Difficulty of multicast protocols (like PTP) over WiFi.
  - What do the networking standards say?
    - IEEE Std 802.11-2016
      - Timing Measurement (TM)
      - Fine Timing Measurement (FTM)
    - IEEE 1588
      - Media-Dependent, Media-Independent MDMI
      - Special Ports
    - 802.1AS
      - Fine Timing Measurement Burst
  - Which of the above can be used for a practical solution?
    - What are the advantages/disadvantages of TM versus FTM?
    - What alternatives might we pursue?
      - unicast PTP without FTM
      - AP as transparent clock
- Existing Linux interfaces for time synchronization
  - What can be used as is?
  - What new interaces or extensions are needed, and why?
- Vendor support
  - How will we encourage broad acceptance/coverage?

IMO, the simplest way that will unlock many use cases is to provide time stamps for single unicast frames, like in ieee80211_rx_status.device_timestamp and expose an adjustable PHC using timecounter/cyclecounter over the free running usec clock.  Then you could synchronize client/AP over unicast IPv4 PTP (for example) with no user space changes needed, AND it would work on all radios, even those that don't implement FTM.

Thanks,
Richard
---------------------------------------------------------------------
A member of the Intel Corporation group of companies

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Powered by blists - more mailing lists