Date:   Thu, 10 Nov 2022 10:52:31 -0800
From:   Stanislav Fomichev <sdf@...gle.com>
To:     John Fastabend <john.fastabend@...il.com>
Cc:     bpf@...r.kernel.org, ast@...nel.org, daniel@...earbox.net,
        andrii@...nel.org, martin.lau@...ux.dev, song@...nel.org,
        yhs@...com, kpsingh@...nel.org, haoluo@...gle.com,
        jolsa@...nel.org, David Ahern <dsahern@...il.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Willem de Bruijn <willemb@...gle.com>,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        Anatoly Burakov <anatoly.burakov@...el.com>,
        Alexander Lobakin <alexandr.lobakin@...el.com>,
        Magnus Karlsson <magnus.karlsson@...il.com>,
        Maryam Tahhan <mtahhan@...hat.com>, xdp-hints@...-project.net,
        netdev@...r.kernel.org
Subject: Re: [RFC bpf-next v2 04/14] veth: Support rx timestamp metadata for xdp

On Thu, Nov 10, 2022 at 9:39 AM John Fastabend <john.fastabend@...il.com> wrote:
>
> Stanislav Fomichev wrote:
> > On Wed, Nov 9, 2022 at 5:35 PM John Fastabend <john.fastabend@...il.com> wrote:
> > >
> > > Stanislav Fomichev wrote:
> > > > On Wed, Nov 9, 2022 at 4:26 PM John Fastabend <john.fastabend@...il.com> wrote:
> > > > >
> > > > > Stanislav Fomichev wrote:
> > > > > > xskxceiver conveniently setups up veth pairs so it seems logical
> > > > > > to use veth as an example for some of the metadata handling.
> > > > > >
> > > > > > We timestamp skb right when we "receive" it, store its
> > > > > > pointer in new veth_xdp_buff wrapper and generate BPF bytecode to
> > > > > > reach it from the BPF program.
> > > > > >
> > > > > > This largely follows the idea of "store some queue context in
> > > > > > the xdp_buff/xdp_frame so the metadata can be reached out
> > > > > > from the BPF program".
> > > > > >
> > > > >
> > > > > [...]
> > > > >
> > > > > >       orig_data = xdp->data;
> > > > > >       orig_data_end = xdp->data_end;
> > > > > > +     vxbuf.skb = skb;
> > > > > >
> > > > > >       act = bpf_prog_run_xdp(xdp_prog, xdp);
> > > > > >
> > > > > > @@ -942,6 +946,7 @@ static int veth_xdp_rcv(struct veth_rq *rq, int budget,
> > > > > >                       struct sk_buff *skb = ptr;
> > > > > >
> > > > > >                       stats->xdp_bytes += skb->len;
> > > > > > +                     __net_timestamp(skb);
> > > > >
> > > > > Just getting to reviewing in depth a bit more. But we hit veth with lots of
> > > > > packets in some configurations I don't think we want to add a __net_timestamp
> > > > > here when vast majority of use cases will have no need for timestamp on veth
> > > > > device. I didn't do a benchmark but its not free.
> > > > >
> > > > > If there is a real use case for timestamping on veth we could do it through
> > > > > a XDP program directly? Basically fallback for devices without hw timestamps.
> > > > > Anyways I need the helper to support hardware without time stamping.
> > > > >
> > > > > Not sure if this was just part of the RFC to explore BPF programs or not.
> > > >
> > > > Initially I've done it mostly so I can have selftests on top of veth
> > > > driver, but I'd still prefer to keep it to have working tests.
> > >
> > > I can't think of a use for it though so its just extra cycles. There
> > > is a helper to read the ktime.
> >
> > As I mentioned in another reply, I wanted something SW-only to test
> > this whole metadata story.
>
> Yeah I see the value there. Also because this is in the veth_xdp_rcv
> path we don't actually attach XDP programs to veths except for in
> CI anyways. I assume though if someone actually does use this in
> prod having an extra _net_timestamp there would be extra unwanted
> cycles.
>
> > The idea was:
> > - veth rx sets skb->tstamp (otherwise it's 0 at this point)
> > - veth kfunc to access rx_timestamp returns skb->tstamp
> > - xsk bpf program verifies that the metadata is non-zero
> > - the above shows end-to-end functionality with a software driver
>
> Yep 100% agree very handy for testing just not sure we can add code
> to hotpath just for testing.
>
> >
> > > > Any way I can make it configurable? Is there some ethtool "enable tx
> > > > timestamping" option I can reuse?
> > >
> > > There is a -T option for timestamping in ethtool. There are also the
> > > socket controls for it. So you could spin up a socket and use it.
> > > But that is a bit broken as well I think it would be better if the
> > > timestamp came from the receiving physical nic?
> > >
> > > I have some mlx nics here and a k8s cluster with lots of veth
> > > devices so I could think a bit more. I'm just not sure why I would
> > > want the veth to timestamp things off hand?
> >
> > -T is for dumping only it seems?
> >
> > I'm probably using skb->tstamp in an unconventional manner here :-/
> > Do you know if enabling timestamping on the socket, as you suggest,
> > will get me some non-zero skb_hwtstamps with xsk?
> > I need something to show how the kfunc can return this data and how
> > can this data land in xdp prog / af_xdp chunk..
>
> Take a look at ./Documentation/networking/timestamping.rst the 3.1
> section is maybe relevant. But then you end up implementing a bunch
> of random ioctls for no reason other than testing. Maybe worth doing
> though for this not sure.

Hmm, there is a call to skb_tx_timestamp in veth_xmit that I missed.
Let me see if I can make it insert skb->tstamp by turning on one of
the timestamping options you mentioned..


> Using virtio driver might be actual useful and give you a test device.
> Early XDP days I used it for testing a lot. Would require qemu to
> setup though.

Powered by blists - more mailing lists