Date: Thu, 15 Feb 2024 08:38:39 +0000
From: Ferenc Fejes <ferenc.fejes@...csson.com>
To: "kurt@...utronix.de" <kurt@...utronix.de>, "netdev@...r.kernel.org"
	<netdev@...r.kernel.org>
CC: "vinicius.gomes@...el.com" <vinicius.gomes@...el.com>, "hawk@...nel.org"
	<hawk@...nel.org>
Subject: Re: igc: AF_PACKET and SO_TXTIME question

Hi Kurt,

On Wed, 2024-02-14 at 15:51 +0100, Kurt Kanzenbach wrote:
> Hi Ferenc,
> 
> On Wed Feb 14 2024, Ferenc Fejes wrote:
> > Hi,
> > 
> > We are experimenting with scheduled packet transfers using the
> > AF_PACKET socket. There is the ETF disk, which is great for most
> > cases.
> 
> For what use cases does it not work? Are you running PREEMPT_RT? Just
> asking out of curiosity :-).

For some experiment, I need control over when the packets drained from
the ETF queue to the NIC. Right now if you schedule some packets lets
say 0.5 sec in the future, they are sitting inside the ETF queue for a
while, then when the txtime is near, they pushed to the NIC's buffer. I
want to skip the ETF part to see how the NIC behave on its own. Its a
black box right now, but we want to see some edge case behavior, since
we will have mixed host and AF_PACKET/XDP traffic in our experiment. 

> 
> > When we bypassed ETF, everything seemed ok regarding the timing:
> > our
> > packet received about +/-15ns offset at the receiver (now its the
> > same
> > machine just to make sure with the timesync) compared to the
> > timestamp
> > set with SO_TXTIME CMSG.
> > 
> > What we tried now is to bypass the ETF qdisc. We enabled the ETF
> > qdisc
> > with hardware offload and sent the exact same packets, but this
> > time
> > with PACKET_QDISC_BAYPASS enabled on the AF_PACKET socket. The
> > codepath
> > looks good, the qdisc part is not called, the packet_snd calls the
> > dev_direct_xmit which calls the igc_xmit_frame. However, in this
> > case
> > the packet was sent more or less immediately.
> 
> Well, yeah the code path looks good indeed. packet_snd() copies the
> transmit time which is provided by the CMSG and calls into
> packet_xmit(), dev_direct_xmit()...
> 
> > 
> > I wonder why we do not see the delayed sending in this case? We
> > tried
> > with different offsets (e.g. 0.1, 0.01, 0.001 sec in the future)
> > but we
> > received the packet after 20-30usec every time. I cant see any code
> > that touches the skb timestamp after the packet_snd, so I suspect
> > that
> > the igc_xmit_frame sees the same timestamp that it would see in the
> > non-baypass case.
> 
> Maybe add some trace_printk()s to track what timestamps are actually
> calculated in igc_tx_launchtime() and if it makes sense?

Thanks, we looked into with the following script:

bpftrace -e 'kfunc:igc_xmit_frame{printf("%ld\n", args->skb->tstamp);}'

The timestamp looks correct for both cases. We will double check that,
but after a quick test the timestamp exactly what we set with SO_TXTIME
CMSG both qdisc and bypass cases.

> 
> Second point to make sure is that the Tx queue your packet is being
> transmitted to has Launch Time enabled.

Good point! We missed that. IIUC the TX queue selected by the skb-
>queue_mapping after the igc_xmit_frame part. We set that parameter by
skbedit tc  action but that is not executed in the bypass case. For
that reason, the skb->queue_mapping more or less random.

However, for a quick test, we set ETF offload for all 4 TX queues. So
no matter which queue the packet enqueued it should respect the
launchtime. But we dont really see this behavior... Nevertheless its a
good catch, I missed this aspect completely at the time of my first
email.

> 
> > 
> > I happen to have the i225 user manual, but after some grep I cannot
> > find any debug registers or counters to monitor the behavior of the
> > scheduled transmission (scheduling errors or bad timestamps, etc.).
> > Are
> > there any?
> 
> Not that I'm aware of.
> 
> > 
> > I am afraid this issue might also be relevant for the AF_XDP case,
> > which also hooks after the qdisc layer, so the launchtime (or
> > whatever
> > it is called) is handled directly by the igc driver.
> 
> Is that already possible with AF_XDP? There were some patches on
> xdp-hints, but i don't think it has been merged yet.

Not yet. There was a patchset with the infra and stmmac as an user,
right now I think its under some rework including igc as another user. 
However in principle, the codepath might be similar to AF_PACKET, the
only difference is the lunchtime metadata bundled with the xdp_buff and
not with the skb.

> 
> Thanks,
> Kurt

Best,
Ferenc

Powered by blists - more mailing lists