Message-ID: <CAA93jw5Ab+ZVq48JAcA0s_=fLBJ4OCmVfsuK6=K3A24_k0tyYg@mail.gmail.com>
Date:   Mon, 10 May 2021 08:09:46 -0700
From:   Dave Taht <dave.taht@...il.com>
To:     Frieder Schrempf <frieder.schrempf@...tron.de>
Cc:     NXP Linux Team <linux-imx@....com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>
Subject: Re: i.MX8MM Ethernet TX Bandwidth Fluctuations

On Mon, May 10, 2021 at 5:49 AM Frieder Schrempf
<frieder.schrempf@...tron.de> wrote:
>
> Hi Dave,
>
> thanks for the input. I really don't know much about the networking stack, so at the moment I can only provide the values requested below, without knowing what it really means.
>
> What's so strange is, that the performance is actually good in general and only "snaps" to the "bad" state and back after some time or after repeated test runs.
>
> And by the way, the ethernet driver in use is the FEC driver at drivers/net/ethernet/freescale/fec_main.c.

It doesn't look (from a quick grep) that that driver ever got BQL support.

davet@...rges-MacBook-Pro freescale % grep sent_queue *.c
gianfar.c:    netdev_tx_sent_queue(txq, bytes_sent);
ucc_geth.c:    netdev_sent_queue(dev, skb->len);

If you really care about bidirectional throughput, having enormous
fifo buffers buried deep in the driver has a tendency to hurt that a
lot and has the symptoms you describe, however not as persistent, so I
would suspect another bug involving gso or gro to start with...

BUT: I note that the effort in implementing bql and testing the packet
size accounting usually shows up other problems in the tx/rx ring, GRO
or NAPI code, and thus is worthwhile exercise that might find where
things are getting stuck.

It doesn't appear your kernel has fq_codel qdisc support, either,
which means big dumb fifos at that layer, drastically affecting bidir
throughput. also.

Since the nxp team is cc'd this is a preso I'd given broadcom back in 2018:

http://www.taht.net/~d/broadcom_aug9_2018.pdf

And the relevant lwn articles from, like, 2011:

https://lwn.net/Articles/454390/

https://lwn.net/Articles/469652/


If someone wants to send me a board to play with...

> On 06.05.21 16:53, Dave Taht wrote:
> > I am a big fan of bql - is that implemented on this driver?
> >
> > cd /sys/class/net/your_device_name/queues/tx-0/byte_queue_limits/
> > cat limit
>
> ~# cat /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit
> 0
>
> >
> > see also bqlmon from github
> >
> > is fq_codel running on the ethernet interface? the iperf bidir test
> > does much better with that in place rather than a fifo. tc -s qdisc
> > show dev your_device
>
> ~# tc -s qdisc show dev eth0
> RTNETLINK answers: Operation not supported
> Dump terminated
>
> Best regards
> Frieder
>
> >
> > Also I tend to run tests using the flent tool, which will yield more
> > data. Install netperf and irtt on the target, flent, netperf, irtt on
> > the test driver box...
> >
> > flent -H the-target-ip -x --socket-stats -t whateveryouaretesting rrul
> > # the meanest bidir test there
> >
> > flent-gui *.gz
> >
> > On Thu, May 6, 2021 at 7:47 AM Frieder Schrempf
> > <frieder.schrempf@...tron.de> wrote:
> >>
> >> Hi,
> >>
> >> we observed some weird phenomenon with the Ethernet on our i.MX8M-Mini boards. It happens quite often that the measured bandwidth in TX direction drops from its expected/nominal value to something like 50% (for 100M) or ~67% (for 1G) connections.
> >>
> >> So far we reproduced this with two different hardware designs using two different PHYs (RGMII VSC8531 and RMII KSZ8081), two different kernel versions (v5.4 and v5.10) and link speeds of 100M and 1G.
> >>
> >> To measure the throughput we simply run iperf3 on the target (with a short p2p connection to the host PC) like this:
> >>
> >>         iperf3 -c 192.168.1.10 --bidir
> >>
> >> But even something more simple like this can be used to get the info (with 'nc -l -p 1122 > /dev/null' running on the host):
> >>
> >>         dd if=/dev/zero bs=10M count=1 | nc 192.168.1.10 1122
> >>
> >> The results fluctuate between each test run and are sometimes 'good' (e.g. ~90 MBit/s for 100M link) and sometimes 'bad' (e.g. ~45 MBit/s for 100M link).
> >> There is nothing else running on the system in parallel. Some more info is also available in this post: [1].
> >>
> >> If there's anyone around who has an idea on what might be the reason for this, please let me know!
> >> Or maybe someone would be willing to do a quick test on his own hardware. That would also be highly appreciated!
> >>
> >> Thanks and best regards
> >> Frieder
> >>
> >> [1]: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.nxp.com%2Ft5%2Fi-MX-Processors%2Fi-MX8MM-Ethernet-TX-Bandwidth-Fluctuations%2Fm-p%2F1242467%23M170563&amp;data=04%7C01%7Cfrieder.schrempf%40kontron.de%7C157b00b2686447fd9a7108d9109ecbc6%7C8c9d3c973fd941c8a2b1646f3942daf1%7C0%7C0%7C637559096478620665%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=SFT%2Boic9C1sirw%2BT1o1qRNNUe4H9bk2FHkLQpdy489I%3D&amp;reserved=0
> >
> >
> >



-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

Powered by blists - more mailing lists