Date:   Wed, 16 Oct 2019 13:02:57 -0400
From:   Murali Karicheri <m-karicheri2@...com>
To:     Vinicius Costa Gomes <vinicius.gomes@...el.com>,
        Vladimir Oltean <olteanv@...il.com>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: taprio testing - Any help?

Hi Vinicius,

On 10/14/2019 07:39 PM, Vinicius Costa Gomes wrote:
> Murali Karicheri <m-karicheri2@...com> writes:
>>
>> My expectation is as follows
>>
>> AAAAAABBBBBCCCCCDDDDDEEEEE
>>
>> Where AAAAA is traffic from TC0, BBBBB is udp stream for port 10000
>> CCCCC is stream for port 20000, DDDDD for 30000 and EEEEE for 40000.
>> Each can be max of 4 msec. Is the expection correct? At least that
>> is my understanding.
> 
> Your expectation is correct.
> 
>>
>> But what I see is alternating packets with port 10000/20000/30000/40000
>> at the wireshark capture and it doesn't make sense to me. If you
>> look at the timestamp, there is nothing showing the Gate is honored
>> for Tx. Am I missing something?
> 
> Remember that taprio (in software mode) has no control after the packet
> is delivered to the driver. So, even if taprio obeys your traffic
> schedule perfectly, the driver/controller may decide to send packets
> according to some other logic.
> 
That is true.

I think I get why it can't work without ETF offload which is missing in
our hardware. Here is what my understanding. Please correct it if wrong.

Our hardware has priority queues implemented. So if there are no
packets in the higher priority queue, it would send from the lower
priority ones. Assuming packets gets dequeue-ed correctly by
taprio and that packets are only in one of the lower priority TC.
i.e in the above example, BBBBBB are present when TC1 Gate is open.
Assuming there are more packets than actually sent out during TC1
window, and assuming no packets in the TC0 queue (AAAAA is absent)
then hardware will continue to send from TC1 queue. So that might
be what is happening, right?

So it is required to deliver frames to driver only when the Gate for
the specific traffic class is open. Is that what is done by ETF qdisc?
 From ETF description at
http://man7.org/linux/man-pages/man8/tc-etf.8.html
'The ETF (Earliest TxTime First) qdisc allows applications to control
the instant when a packet should be dequeued from the traffic control
layer into the netdevice'. So I assume, when I use iperf (there is
no txtime information in the packet), I still can use ETF and
packet time will be modified to match with schedule and then get
dequeue-ed at correct time to arrive at the driver during the Gate
open of taprio. Is this correct?

If ETF can schedule packet to arrive at the driver just during th
Gate open and work in sync with taprio scheduler, that would do the
work.I understand the border may be difficult to manage. However if we
add a guard band by adding an extra entry with all Gates closed
between schedules for guard band duration, it should allow hardware to
flush out any remaining frames from the queue outside its Gate duration.
If my understanding is correct, can I use software ETF qdisc in this
case? If so how do I configure it? Any example?

>>
>> The tc stats shows packets are going through specific TC/Gate
>>
>> root@...7xx-evm:~# tc -d -p -s qdisc show dev eth0
>> qdisc taprio 100: root refcnt 9 tc 5 map 0 1 2 3 4 4 4 4 4 4 4 4 4 4 4 4
>> queues offset 0 count 1 offset 1 count 1 offset 2 count 1 offset 3 count
>> 1 offset 4 count 1
>> clockid TAI offload 0   base-time 0 cycle-time 0 cycle-time-extension 0
>> base-time 1564768921123459533 cycle-time 20000000 cycle-
>> time-extension 0
>>           index 0 cmd S gatemask 0x1 interval 4000000
>>           index 1 cmd S gatemask 0x2 interval 4000000
>>           index 2 cmd S gatemask 0x4 interval 4000000
>>           index 3 cmd S gatemask 0x8 interval 4000000
>>           index 4 cmd S gatemask 0x10 interval 4000000
>>
>>    Sent 80948029 bytes 53630 pkt (dropped 0, overlimits 0 requeues 0)
>>    backlog 0b 0p requeues 0
>> qdisc pfifo 0: parent 100:5 limit 1000p
>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>    backlog 0b 0p requeues 0
>> qdisc pfifo 0: parent 100:4 limit 1000p
>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>    backlog 0b 0p requeues 0
>> qdisc pfifo 0: parent 100:3 limit 1000p
>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>    backlog 0b 0p requeues 0
>> qdisc pfifo 0: parent 100:2 limit 1000p
>>    Sent 16184448 bytes 10704 pkt (dropped 0, overlimits 0 requeues 0)
>>    backlog 0b 0p requeues 0
>> qdisc pfifo 0: parent 100:1 limit 1000p
>>    Sent 16210237 bytes 10814 pkt (dropped 0, overlimits 0 requeues 0)
>>    backlog 0b 0p requeues 0
>>
>> Also my hardware queue stats shows frames going through correct queues.
>> Am I missing something?
>>
> 
> What I usually see in these cases, are that the borders (from A to B,
> for example) are usually messy, the middle of each entry are more well
> behaved.
> 
OK

> But there are things that could improve the behavior: reducing TX DMA
> coalescing, reducing the number of packet buffers in use in the
> controller, disabling power saving features, that kind of thing.
I can try playing with the number if descriptors used. But from my above
response, I might have to use software ETF qdisc along with taprio to
have packets in the correct order on the wire. So will wait on this for
now.
> 
> If you are already doing something like this, then I would like to know
> more, that could indicate a problem.
> 
No. The hardware just implement a priority queue scheme. I can control
the number of buffers or descriptors.

Murali
> [...]
> 
>> I am on a 4.19.y kernel with patches specific to taprio
>> backported. Am I missing anything related to taprio. I will
>> try on the latest master branch as well. But if you can point out
>> anything that will be helpful.
>>
> 
> [...]
> 
>> lcpd/ti-linux-4.19.y) Merged TI feature connectivity into
>> ti-linux-4.19.y
> 
> I can't think of anything else.
> 
>>
>>>
>>> Regards,
>>> -Vladimir
>>>
> 
> Cheers,
> --
> Vinicius
> 

Powered by blists - more mailing lists