Message-ID: <a4a8adc8-4d4c-3b09-6c2f-ce1d12e0b9bc@nvidia.com>
Date:   Sun, 6 Dec 2020 15:37:47 +0200
From:   Eran Ben Elisha <eranbe@...dia.com>
To:     Jakub Kicinski <kuba@...nel.org>, Saeed Mahameed <saeed@...nel.org>
CC:     "David S. Miller" <davem@...emloft.net>, <netdev@...r.kernel.org>,
        "Tariq Toukan" <tariqt@...dia.com>
Subject: Re: [net-next V2 08/15] net/mlx5e: Add TX PTP port object support



On 12/5/2020 2:24 AM, Jakub Kicinski wrote:
> On Fri, 04 Dec 2020 15:57:36 -0800 Saeed Mahameed wrote:
>> On Fri, 2020-12-04 at 15:17 -0800, Jakub Kicinski wrote:
>>> On Fri, 04 Dec 2020 13:57:49 -0800 Saeed Mahameed wrote:
>>>>>> option 2) route PTP traffic to a special SQs per ring, this SQ
>>>>>> will
>>>>>> be
>>>>>> PTP port accurate, Normal traffic will continue through regular
>>>>>> SQs
>>>>>>
>>>>>> Pros: Regular non PTP traffic not affected.
>>>>>> Cons: High memory footprint for creating special SQs
>>>>>>
>>>>>> So we prefer (2) + private flag to avoid the performance hit
>>>>>> and
>>>>>> the
>>>>>> redundant memory usage out of the box.
>>>>>
>>>>> Option 3 - have only one special PTP queue in the system. PTP
>>>>> traffic
>>>>> is rather low rate, queue per core doesn't seem necessary.
>>>>
>>>> We only forward ptp traffic to the new special queue but we create
>>>> more
>>>> than one to avoid internal locking as we will utilize the tx
>>>> softirq
>>>> percpu.
>>>
>>> In other words to make the driver implementation simpler we'll have
>>> a pretty basic feature hidden behind a ethtool priv knob and a number
>>> of queues which doesn't match reality reported to user space. Hm.
>>
>> I look at these queues as a special HW objects to allow the accurate
>> PTP stamping, they piggyback on the reported txqs, so they are
>> transparent,
> 
> But they are visible to the stack, via sysfs, netlink. Any check
> in the kernel that tries to help the driver by validating user input
> against real_num_tx_queues will be moot for mlx5e.

Re-writing it here,  we report them in real num of TX queues.

> 
> mlx5e hides the AF_XDP queues behind normal RSS queues, but it would
> have extra visible queues for TX PTP.
> 
>> they just increase the memory footprint of each ring.
> 
> For every ring or for every TC? (which is hopefully 1 in any non-DCB
> deployment?)

For every TC, not for every ring.

> 
>> for the priv flags, one of the floating ideas was to
>> use hwtstamp_rx_filters flags:
>>   
>> https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/net_tstamp.h#L107
>>
>> Our hardware timestamps all packets for free whether you request it or
>> not, Currently there is no option to setup "ALL_PTP" traffic in ethtool
>> -T, but we can add this flag as it make sense to be in ethtool -T, thus
>> we could use it in mlx5 to determine if user selected ALL_PTP, then ptp
>> packets will go through this accurate special path.
>>
>> This is not a W/A or an abuse to the new flag, it just means if you
>> select ALL_PTP then a side effect will be our HW will be more accurate
>> for PTP traffic.
>>
>> What do you think ?
> 
> That sounds much better than the priv flag, yes.

Our Hardware can provide a better accurate time stamp under few 
limitations. It requires higher memory consumption ({SQ, 2 x CQ, 
internal HW LB RQ} per TC), and also has performance impact (more CQEs 
to consume for example).
Some customers are happy with the accuracy they get today and don't want 
the extra penalty, so they don't want to be automatically shifted to the 
new TS logic.

Adding new enum to the ioctl means we have add 
(HWTSTAMP_TX_ON_TIME_CRITICAL_ONLY for example) all the way - drivers, 
kernel ptp, user space ptp, ethtool.

My concerns are:
1. Timestamp applications (like ptp4l or similar) will have to add 
support for configuring the driver to use 
HWTSTAMP_TX_ON_TIME_CRITICAL_ONLY if supported via ioctl prior to 
packets transmit. From application point of view, the dual-modes 
(HWTSTAMP_TX_ON_TIME_CRITICAL_ONLY , HWTSTAMP_TX_ON) support is 
redundant, as it offers nothing new.
2. Other vendors will have to support it as well, when not sure what is 
the expectation from them if they cannot improve accuracy between them.

This feature is just an internal enhancement, and as such it should be 
added only as a vendor private configuration flag. We are not offering 
here about any standard for others to follow.

If we did not have the limitation above, it could have been added as the 
default silently.

I suggest we reconsider the ethtool private-flag, the ioctl change might 
be a long journey in a wrong direction.

> 
>> Regarding reducing to a single special queue, i will discuss with Eran
>> and the Team on Sunday.
> 
> Okay, thanks.
> 

Powered by blists - more mailing lists