Date:   Wed, 11 Nov 2020 22:30:33 +0000
From:   "Patel, Vedang" <vedang.patel@...el.com>
To:     Jesper Dangaard Brouer <brouer@...hat.com>
CC:     Andrii Nakryiko <andrii.nakryiko@...il.com>,
        David Ahern <dsahern@...nel.org>,
        Ilias Apalodimas <ilias.apalodimas@...aro.org>,
        "Saeed Mahameed" <saeed@...nel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "Gomes, Vinicius" <vinicius.gomes@...el.com>,
        "Guedes, Andre" <andre.guedes@...el.com>
Subject: Re: Hardware time stamping support for AF_XDP applications

[Looks like format settings for my mail client were not persistent and I sent out HTML message again by mistake. Hopefully it’s fixed now. Sorry for the spam. :( ]

Hi Jesper, 

Thanks for all the inputs. My Responses inline:

> On Nov 11, 2020, at 6:53 AM, Jesper Dangaard Brouer <brouer@...hat.com> wrote:
> 
> On Tue, 10 Nov 2020 23:53:41 +0000
> "Patel, Vedang" <vedang.patel@...el.com> wrote:
> 
>> Hi Saeed, 
>> 
>>> On Nov 10, 2020, at 3:32 PM, Saeed Mahameed <saeed@...nel.org> wrote:
>>> 
>>> On Tue, 2020-11-10 at 22:44 +0000, Patel, Vedang wrote:  
>>>> [Sorry if you got the email twice. Resending because it was rejected
>>>> by netdev for containing HTML]
>>>> 
>>>> Hi Saeed/Jesper, 
>>>> 
>>>> I am working in the Time Sensitive Networking team at Intel. We work
>>>> on implementing and upstreaming support for TSN related features for
>>>> intel based NICs. Recently we have been adding support for XDP in
>>>> i225. One of the features which we want to add support for is passing
>>>> the hardware timestamp information to the userspace application
>>>> running AF_XDP sockets (for both Tx and Rx). I came across the XDP
>>>> Workshop[1] conducted in July 2020 and there you stated that you are
>>>> already working on adding support for BTF based metadata to pass
>>>> hardware hints for XDP Programs. My understanding (along with a few
>>>> questions) of the current state is:  
> 
> Have the i225 XDP support been upstreamed?
Andre (CC’d here) is working on adding support. The first series [1] just got merged into dev queue. He is working on adding Zero Copy and UMEM support.

[1] - https://patchwork.ozlabs.org/project/intel-wired-lan/cover/20201104031210.27772-1-andre.guedes@intel.com/
> 
> Can I buy a i255 NIC for my server, or is this embedded NICs?
Yes, i225 NICs are available in discrete form factor. One of the links where it’s publicly available is:
https://shop.qnap.com/index.php?route=product/product&product_id=388

> 
> Ilias have played with PoC for TSN (on ARM) here:
> https://github.com/xdp-project/xdp-project/blob/master/areas/arm64/xdp_for_tsn.org
Our aim is to use the approach described in the last paragraph to gather end-to-end latency.
> 
>>> Hi Patel,
>>> 
>>>> * This feature is currently being maintained out of tree. I found
>>>> that an RFC Series[2] was posted in June 2018. Are you planning to
>>>> post an updated version to be merged in the mainline anytime soon?   
>>> 
>>> Yes hopefully in the coming couple of weeks.
>>> 
>> 
>> Sure! I will start testing/developing on top of your branch mentioned
>> below for now.
> 
> I've also signed up for helping out on this effort.  Notice Andrii (cc)
> have already pointed out something that can be improved, and even made
> easier.
That’s great! Can you point me to the improvement Andrii mentioned?
> 
> 
>>>> * I am guessing hardware timestamp is one of the metadata fields
>>>> which will be eventually supported? [3]  
>>> 
>>> With BTF formatted metadata it is up to the driver to advertise
>>> whatever it can/want :)
>>> so yes.  
>> 
>> I have a very basic question here. From what I understand about BTF,
>> I can generate a header file (using bpftool?) containing the BTF data
>> format provided by the driver. If so, how can I design an application
>> which can work with multiple NICs drivers without recompilation? I am
>> guessing there is some sort of “master list” of HW hints the drivers
>> will agree upon?
> 
> I recommend that you read Andrii's blogpost:
> https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
This is quite insightful!
> 
> I need to learn more about BTF myself, but the way I understand this:
> We need to agree on the meaning of struct member names (e.g. rxhash32).
> Then you compile BPF with a BTF struct that have this rxhash32 member
> name, and at BPF load-time the kernel CO-RE infra will remap the offset
> to the rxhash32 offset used by the specific driver.
> 
>>> 
>>>> * The Metadata support will be extended to pass on the hardware hints
>>>> to AF_XDP sockets. Are there any rough plans on what metadata will be
>>>> transferred?  
>>> 
>>> AF_XDP is not part of my series, but supporting AF_XDP with metadata
>>> offlaod is up to the driver to implement, should be straight forward
>>> and identical to XDP.
> 
> The XDP data_meta area is also transferred into the AF_XDP frame, also
> in the copy-mode version of AF_XDP.
> 
> 
>>> what meta data to pass is up to the driver.  
>> 
>> Alright, let me take a closer look at your latest code. I will come
>> back will questions if I have any.
>>> 
>>> 
>>>> * The current plan for Tx side only includes passing data from the
>>>> application to the driver. Are there any plans to support passing
>>>> information (like HW TX timestamp) from driver to the Application?
>>>> 
>>> 
>>> you mean for AF_XDP ? i actually haven't thought about this, 
>>> but we could use TX umem packet buffer headroom to pass TX completion
>>> metadata to AF_XDP app, or extend the completion queue entries to host
>>> metadata, i am sure that the 1st approach is preferred, but i am not
>>> planing to support this in my initial series. 
>>> 
>> Yeah, I was thinking of using approach 1 as well for this. Let me
>> first work on the Rx side. Then we can scope this one out.
>> 
>>>> Finally, is there any way I can help in expediting the development
>>>> and upstreaming of this feature? I have been working on studying how
>>>> XDP works and can work on implementing some part of this feature if
>>>> you would like.
>>>> 
>>> 
>>> Sure,
>>> Please feel free to clone and test the following branch if you add
>>> support to  your driver and implement offloads for AF_XDP that would be
>>> awesome, and i will append your patches to my series before submission.
>>> 
>>> it is always great to send new features with multiple use cases and
>>> multi vendor support, this will differently expedite submission and
>>> acceptance
>>> 
>>> My Latest work can be found at:
>>> 
>>> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/log/?h=topic/xdp_metadata3
>>> 
>>> Please feel free to send me any questions about the code in private or
>>> public.
>> 
>> Thanks Saeed for all the information! This is really helpful. :)
>>> 
>>> Thanks,
>>> Saeed.
>>> 
>>>> Thanks,
>>>> Vedang Patel
>>>> Software Engineer
>>>> Intel Corporation
>>>> 
>>>> [1] - https://netdevconf.info/0x14/session.html?workshop-XDP
>>>> [2] - 
>>>> https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
>>>> [3] - 
>>>> https://xdp-project.net/#outline-container-Important-medium-term-tasks  
>> 
> 
> 
> 
> -- 
> Best regards,
>  Jesper Dangaard Brouer
>  MSc.CS, Principal Kernel Engineer at Red Hat
>  LinkedIn: http://www.linkedin.com/in/brouer

Powered by blists - more mailing lists