Date:	Thu, 26 Mar 2009 16:30:10 +0100
From:	Patrick Ohly <patrick.ohly@...el.com>
To:	Herbert Xu <herbert@...dor.apana.org.au>
Cc:	David Miller <davem@...emloft.net>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"Kirsher, Jeffrey T" <jeffrey.t.kirsher@...el.com>
Subject: Re: TX time stamping

On Thu, 2009-03-26 at 14:48 +0000, Herbert Xu wrote:
> On Wed, Mar 25, 2009 at 08:08:44AM +0100, Patrick Ohly wrote:
> >
> > I suggest to make it so that the sender gets the packet back once per
> > interface, with different time stamps and information about the
> > interface.
> 
> That could be awkward.  What if the second or subsequent instance
> of the packet is held up indefinitely?

The sender must be prepared for packet loss anyway (for example, during
stress tests I have seen that packets were dropped between socket layer
and device driver).

Getting a TX time stamp only for some, but not all expected interfaces,
or getting some results much later isn't that different.

After a certain timeout the sender must assume that the packet was lost
and resend. An unexpected response for a packet that was supposed to be
lost must be ignored.

> > > As to the problem of skb->sk, I don't think that's even needed
> > > as we can simply use the skb shared area as the communication
> > > medium.
> > 
> > Can you elaborate on that?
> 
> I was thinking of the sender keeping hold of the sent skb and
> using its shared area to retrieve the time stamp(s).

The sender is in user space, at least in the case of PTP. Given the
current user space APIs it seems complicated to tie the sender to a
particular skb :-/

> > The drawback is the more costly socket lookup. For PTP this isn't an
> > issue due to the low packet rate, so a very simple solution would be
> > good enough. But for other use cases it might be problematic. I also
> > have no idea how the locking for the socket lookup can be done safely.
> 
> Could you give the main usage scenarios for timestamps, including
> what the user-space app does with the info?

The main usage scenario right now is a hardware assisted implementation
of the Precision Time Protocol (PTP, IEEE 1588). "main usage" as in "the
one I worked on"; others have expressed interest in this hardware
assisted time stamping for other purposes, including detailed network
monitoring.

With PTP, the user space daemons exchange multicast messages with a rate
of less than one per second. Sender and receiver record send and receive
time stamps of each packet. This is done in both directions. These two
delays are used to determine clock drift between two nodes. Filters are
used to remove noise in the input data as much as possible, but it is
much better to avoid the noise in the first place.

Noise in this context are varying delays of the packets. It's okay if
the delay is large, as long as it is constant. But when the delay
increases by a value of x compared to the previous measurement, then PTP
cannot tell whether this particular packet just was transmitted more
slowly or the clocks have drifted apart.

Hardware time stamping removes the varying packet delays inside the
kernel's IP stack because the send/receive time stamps are generated
directly when the packet hits the wire. I found that hardware assistance
is essential for good accuracy [1].

It's also important to remove the varying delays inside the network
switches from the equation. PTP v2 has support for that and equipment
manufacturers are starting to support it.

[1] http://www.linuxclustersinstitute.org/conferences/archive/2008/PDF/Ohly_92221.pdf

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists