Date:   Thu, 17 Oct 2019 09:56:51 -0400
From:   Murali Karicheri <m-karicheri2@...com>
To:     Vinicius Costa Gomes <vinicius.gomes@...el.com>,
        Vladimir Oltean <olteanv@...il.com>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: taprio testing - Any help?

Hi Vinicius,
On 10/16/2019 04:32 PM, Vinicius Costa Gomes wrote:
> Murali Karicheri <m-karicheri2@...com> writes:
> 
>> Hi Vinicius,
>>
>> On 10/14/2019 07:39 PM, Vinicius Costa Gomes wrote:
>>> Murali Karicheri <m-karicheri2@...com> writes:
>>>>
>>>> My expectation is as follows
>>>>
>>>> AAAAAABBBBBCCCCCDDDDDEEEEE
>>>>
>>>> Where AAAAA is traffic from TC0, BBBBB is udp stream for port 10000
>>>> CCCCC is stream for port 20000, DDDDD for 30000 and EEEEE for 40000.
>>>> Each can be max of 4 msec. Is the expection correct? At least that
>>>> is my understanding.
>>>
>>> Your expectation is correct.
>>>
>>>>
>>>> But what I see is alternating packets with port 10000/20000/30000/40000
>>>> at the wireshark capture and it doesn't make sense to me. If you
>>>> look at the timestamp, there is nothing showing the Gate is honored
>>>> for Tx. Am I missing something?
>>>
>>> Remember that taprio (in software mode) has no control after the packet
>>> is delivered to the driver. So, even if taprio obeys your traffic
>>> schedule perfectly, the driver/controller may decide to send packets
>>> according to some other logic.
>>>
>> That is true.
>>
>> I think I get why it can't work without ETF offload which is missing in
>> our hardware. Here is what my understanding. Please correct it if
>> wrong.
> 
> For taprio, to get good results, you have to have some kind of
> offloading, so right now, there are two alternatives for offloading: (1)
> full offloading, something similar to what Vladimir added for the
> SJA1105; (2) txtime-assisted mode, what Vedang added to support running
> Qbv-like schedules in controllers that only support controlling the
> transmission time of individual packets (the LaunchTime feature of the
> i210 controller, for example).
> 
> If your hardware doesn't have any of those capabilities, then you are
> basically stuck with the software mode, or you can come up with some
> other "assisted mode" that might work for your hardware.
> 
>>
>> Our hardware has priority queues implemented. So if there are no
>> packets in the higher priority queue, it would send from the lower
>> priority ones. Assuming packets gets dequeue-ed correctly by
>> taprio and that packets are only in one of the lower priority TC.
>> i.e in the above example, BBBBBB are present when TC1 Gate is open.
>> Assuming there are more packets than actually sent out during TC1
>> window, and assuming no packets in the TC0 queue (AAAAA is absent)
>> then hardware will continue to send from TC1 queue. So that might
>> be what is happening, right?
>>
>> So it is required to deliver frames to driver only when the Gate for
>> the specific traffic class is open. Is that what is done by ETF qdisc?
>>   From ETF description at
>> http://man7.org/linux/man-pages/man8/tc-etf.8.html
>> 'The ETF (Earliest TxTime First) qdisc allows applications to control
>> the instant when a packet should be dequeued from the traffic control
>> layer into the netdevice'. So I assume, when I use iperf (there is
>> no txtime information in the packet), I still can use ETF and
>> packet time will be modified to match with schedule and then get
>> dequeue-ed at correct time to arrive at the driver during the Gate
>> open of taprio. Is this correct?
>>
> 
> taprio in the txtime-assisted mode does exactly that: "packet time will
> be modified to match with schedule", but it needs ETF offloading to be
> supported to get good results, ETF has the same "problem" as taprio when
> running in the software mode (no offloading), it has no control after
> the packet is delivered to the driver.
> 
>> If ETF can schedule packet to arrive at the driver just during th
>> Gate open and work in sync with taprio scheduler, that would do the
>> work.I understand the border may be difficult to manage. However if we
>> add a guard band by adding an extra entry with all Gates closed
>> between schedules for guard band duration, it should allow hardware to
>> flush out any remaining frames from the queue outside its Gate duration.
>> If my understanding is correct, can I use software ETF qdisc in this
>> case? If so how do I configure it? Any example?
> 
> Without any offloading, I think you are better off running taprio
> standalone (i.e. without ETF, so you don't have yet another layer of
> packet scheduling based solely on hrtimers), and just adding the
> guard-bands, something like this:
> 
> $ tc qdisc replace dev $IFACE parent root handle 100 taprio \\
>        num_tc 3 \
>        map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
>        queues 1@0 1@1 2@2 \
>        base-time $BASE_TIME \
>        sched-entry S 01 2000000 \
>        sched-entry S 02 3000000 \
>        sched-entry S 04 4000000 \
>        sched-entry S 00 1000000 \
>        clockid CLOCK_TAI
> 
> Thinking a bit more, taprio in txtime-assisted mode and ETF with no
> offloading, *might* be better, if its limitation of only being able to
> use a single TX queue isn't a blocker.
> 
I thought about the same and was playing with ETF with no success so 
far. After I add it it drops all frames. But I was not using flags and
txtime-delay for taprio. So that is the missing part.

So there is a limitation that in this mode only one HW queue can be
specified.
> Something like this:
> 
> $ tc qdisc replace dev $IFACE parent root handle 100 taprio \
>        num_tc 4 \
>        map 2 3 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
>        queues 1@0 1@0 1@0 1@0 \
So here you are specifying all TCs the same Q0. So that is the
limitation you have mentioned above.

>        base-time $BASE_TIME \
>        sched-entry S 0xC 250000 \
>        sched-entry S 0x1 250000 \
>        sched-entry S 0x2 250000 \
>        sched-entry S 0x4 250000 \
>        txtime-delay 300000 \
>        flags 0x1 \
>        clockid CLOCK_TAI
> 
I get an error when I do this in my setup.

root@...7xx-evm:~# tc qdisc replace dev eth0 parent root handle 100 taprio \
 >     num_tc 4 \
 >     map 2 3 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
 >     queues 1@0 1@0 1@0 1@0 \
 >     base-time 1564535762845777831 \
 >     sched-entry S 0xC 15000000 \
 >     sched-entry S 0x2 15000000 \
 >     sched-entry S 0x4 15000000 \
 >     sched-entry S 0x8 15000000 \
 >     txtime-delay 300000 \
 >     flags 0x1 \
 >     clockid CLOCK_TAI
RTNETLINK answers: Invalid argument

Anything wrong with the command syntax?

Thanks and regards,

Murali
> $ tc qdisc replace dev $IFACE parent 100:1 etf \
>        delta 200000 clockid CLOCK_TAI skip_sock_check
> 
> Cheers,
> --
> Vinicius
> 

Powered by blists - more mailing lists