Date:   Wed, 16 Oct 2019 13:22:35 -0400
From:   Murali Karicheri <m-karicheri2@...com>
To:     Vinicius Costa Gomes <vinicius.gomes@...el.com>,
        Vladimir Oltean <olteanv@...il.com>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: taprio testing - Any help?

On 10/16/2019 01:14 PM, Murali Karicheri wrote:
> On 10/16/2019 01:02 PM, Murali Karicheri wrote:
>> Hi Vinicius,
>>
>> On 10/14/2019 07:39 PM, Vinicius Costa Gomes wrote:
>>> Murali Karicheri <m-karicheri2@...com> writes:
>>>>
>>>> My expectation is as follows
>>>>
>>>> AAAAAABBBBBCCCCCDDDDDEEEEE
>>>>
>>>> Where AAAAA is traffic from TC0, BBBBB is udp stream for port 10000
>>>> CCCCC is stream for port 20000, DDDDD for 30000 and EEEEE for 40000.
>>>> Each can be max of 4 msec. Is the expection correct? At least that
>>>> is my understanding.
>>>
>>> Your expectation is correct.
>>>
>>>>
>>>> But what I see is alternating packets with port 10000/20000/30000/40000
>>>> at the wireshark capture and it doesn't make sense to me. If you
>>>> look at the timestamp, there is nothing showing the Gate is honored
>>>> for Tx. Am I missing something?
>>>
>>> Remember that taprio (in software mode) has no control after the packet
>>> is delivered to the driver. So, even if taprio obeys your traffic
>>> schedule perfectly, the driver/controller may decide to send packets
>>> according to some other logic.
>>>
>> That is true.
>>
>> I think I get why it can't work without ETF offload which is missing in
>> our hardware. Here is what my understanding. Please correct it if wrong.
>>
>> Our hardware has priority queues implemented. So if there are no
>> packets in the higher priority queue, it would send from the lower
>> priority ones. Assuming packets gets dequeue-ed correctly by
>> taprio and that packets are only in one of the lower priority TC.
>> i.e in the above example, BBBBBB are present when TC1 Gate is open.
>> Assuming there are more packets than actually sent out during TC1
>> window, and assuming no packets in the TC0 queue (AAAAA is absent)
>> then hardware will continue to send from TC1 queue. So that might
>> be what is happening, right?
>>
>> So it is required to deliver frames to driver only when the Gate for
>> the specific traffic class is open.
> Plus, number of packets delivered should be based on available time
> in the current window.
> 
Also I see in the taprio code

/* There are a few scenarios where we will have to modify the txtime from
  * what is read from next_txtime in sched_entry. They are:
  * 1. If txtime is in the past,
  *    a. The gate for the traffic class is currently open and packet can be
  *       transmitted before it closes, schedule the packet right away.
  *    b. If the gate corresponding to the traffic class is going to 
open later
  *       in the cycle, set the txtime of packet to the interval start.
  * 2. If txtime is in the future, there are packets corresponding to the
  *    current traffic class waiting to be transmitted. So, the following
  *    possibilities exist:
  *    a. We can transmit the packet before the window containing the txtime
  *       closes.
  *    b. The window might close before the transmission can be completed
  *       successfully. So, schedule the packet in the next open window.
  */
static long get_packet_txtime(struct sk_buff *skb, struct Qdisc *sch)
{

So if I enable ETF, looks like packets get dequeue-ed based on txtime
which match with the schedule entries. So packets would arrive at the
driver at the correct time. Of course need to play with delta of the
ETF configuration.

Thanks

Murali
>> Is that what is done by ETF qdisc?
>>  From ETF description at
>> http://man7.org/linux/man-pages/man8/tc-etf.8.html
>> 'The ETF (Earliest TxTime First) qdisc allows applications to control
>> the instant when a packet should be dequeued from the traffic control
>> layer into the netdevice'. So I assume, when I use iperf (there is
>> no txtime information in the packet), I still can use ETF and
>> packet time will be modified to match with schedule and then get
>> dequeue-ed at correct time to arrive at the driver during the Gate
>> open of taprio. Is this correct?
>>
>> If ETF can schedule packet to arrive at the driver just during th
>> Gate open and work in sync with taprio scheduler, that would do the
>> work.I understand the border may be difficult to manage. However if we
>> add a guard band by adding an extra entry with all Gates closed
>> between schedules for guard band duration, it should allow hardware to
>> flush out any remaining frames from the queue outside its Gate duration.
>> If my understanding is correct, can I use software ETF qdisc in this
>> case? If so how do I configure it? Any example?
>>
>>>>
>>>> The tc stats shows packets are going through specific TC/Gate
>>>>
>>>> root@...7xx-evm:~# tc -d -p -s qdisc show dev eth0
>>>> qdisc taprio 100: root refcnt 9 tc 5 map 0 1 2 3 4 4 4 4 4 4 4 4 4 4 
>>>> 4 4
>>>> queues offset 0 count 1 offset 1 count 1 offset 2 count 1 offset 3 
>>>> count
>>>> 1 offset 4 count 1
>>>> clockid TAI offload 0   base-time 0 cycle-time 0 cycle-time-extension 0
>>>> base-time 1564768921123459533 cycle-time 20000000 cycle-
>>>> time-extension 0
>>>>           index 0 cmd S gatemask 0x1 interval 4000000
>>>>           index 1 cmd S gatemask 0x2 interval 4000000
>>>>           index 2 cmd S gatemask 0x4 interval 4000000
>>>>           index 3 cmd S gatemask 0x8 interval 4000000
>>>>           index 4 cmd S gatemask 0x10 interval 4000000
>>>>
>>>>    Sent 80948029 bytes 53630 pkt (dropped 0, overlimits 0 requeues 0)
>>>>    backlog 0b 0p requeues 0
>>>> qdisc pfifo 0: parent 100:5 limit 1000p
>>>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>>>    backlog 0b 0p requeues 0
>>>> qdisc pfifo 0: parent 100:4 limit 1000p
>>>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>>>    backlog 0b 0p requeues 0
>>>> qdisc pfifo 0: parent 100:3 limit 1000p
>>>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>>>    backlog 0b 0p requeues 0
>>>> qdisc pfifo 0: parent 100:2 limit 1000p
>>>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>>>    backlog 0b 0p requeues 0
>>>> qdisc pfifo 0: parent 100:1 limit 1000p
>>>>    Sent 16210237 bytes 10814 pkt (dropped 0, overlimits 0 requeues 0)
>>>>    backlog 0b 0p requeues 0
>>>>
>>>> Also my hardware queue stats shows frames going through correct queues.
>>>> Am I missing something?
>>>>
>>>
>>> What I usually see in these cases, are that the borders (from A to B,
>>> for example) are usually messy, the middle of each entry are more well
>>> behaved.
>>>
>> OK
>>
>>> But there are things that could improve the behavior: reducing TX DMA
>>> coalescing, reducing the number of packet buffers in use in the
>>> controller, disabling power saving features, that kind of thing.
>> I can try playing with the number if descriptors used. But from my above
>> response, I might have to use software ETF qdisc along with taprio to
>> have packets in the correct order on the wire. So will wait on this for
>> now.
>>>
>>> If you are already doing something like this, then I would like to know
>>> more, that could indicate a problem.
>>>
>> No. The hardware just implement a priority queue scheme. I can control
>> the number of buffers or descriptors.
>>
>> Murali
>>> [...]
>>>
>>>> I am on a 4.19.y kernel with patches specific to taprio
>>>> backported. Am I missing anything related to taprio. I will
>>>> try on the latest master branch as well. But if you can point out
>>>> anything that will be helpful.
>>>>
>>>
>>> [...]
>>>
>>>> lcpd/ti-linux-4.19.y) Merged TI feature connectivity into
>>>> ti-linux-4.19.y
>>>
>>> I can't think of anything else.
>>>
>>>>
>>>>>
>>>>> Regards,
>>>>> -Vladimir
>>>>>
>>>
>>> Cheers,
>>> -- 
>>> Vinicius
>>>
>>
>>
> 
> 

Powered by blists - more mailing lists