Message-ID:
 <PAXPR04MB8510C1030E70335B283B4ACE88462@PAXPR04MB8510.eurprd04.prod.outlook.com>
Date: Wed, 16 Oct 2024 02:34:14 +0000
From: Wei Fang <wei.fang@....com>
To: Andrew Lunn <andrew@...n.ch>
CC: Joe Damato <jdamato@...tly.com>, Shenwei Wang <shenwei.wang@....com>,
	"David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>, Clark Wang
	<xiaoning.wang@....com>, "imx@...ts.linux.dev" <imx@...ts.linux.dev>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>, dl-linux-imx
	<linux-imx@....com>
Subject: RE: [PATCH v2 net-next resent] net: fec: Enable SOC specific rx-usecs
 coalescence default setting

> -----Original Message-----
> From: Andrew Lunn <andrew@...n.ch>
> Sent: 2024年8月1日 7:56
> To: Joe Damato <jdamato@...tly.com>; Shenwei Wang
> <shenwei.wang@....com>; Wei Fang <wei.fang@....com>; David S. Miller
> <davem@...emloft.net>; Eric Dumazet <edumazet@...gle.com>; Jakub
> Kicinski <kuba@...nel.org>; Paolo Abeni <pabeni@...hat.com>; Clark Wang
> <xiaoning.wang@....com>; imx@...ts.linux.dev; netdev@...r.kernel.org;
> dl-linux-imx <linux-imx@....com>
> Subject: Re: [PATCH v2 net-next resent] net: fec: Enable SOC specific rx-usecs
> coalescence default setting
> 
> On Tue, Jul 30, 2024 at 11:17:05AM +0100, Joe Damato wrote:
> > On Mon, Jul 29, 2024 at 02:35:27PM -0500, Shenwei Wang wrote:
> > > The current FEC driver uses a single default rx-usecs coalescence setting
> > > across all SoCs. This approach leads to suboptimal latency on newer, high
> > > performance SoCs such as i.MX8QM and i.MX8M.
> > >
> > > For example, the following are the ping result on a i.MX8QXP board:
> > >
> > > $ ping 192.168.0.195
> > > PING 192.168.0.195 (192.168.0.195) 56(84) bytes of data.
> > > 64 bytes from 192.168.0.195: icmp_seq=1 ttl=64 time=1.32 ms
> > > 64 bytes from 192.168.0.195: icmp_seq=2 ttl=64 time=1.31 ms
> > > 64 bytes from 192.168.0.195: icmp_seq=3 ttl=64 time=1.33 ms
> > > 64 bytes from 192.168.0.195: icmp_seq=4 ttl=64 time=1.33 ms
> > >
> > > The current default rx-usecs value of 1000us was originally optimized for
> > > CPU-bound systems like i.MX2x and i.MX6x. However, for i.MX8 and later
> > > generations, CPU performance is no longer a limiting factor. Consequently,
> > > the rx-usecs value should be reduced to enhance receive latency.
> > >
> > > The following are the ping result with the 100us setting:
> > >
> > > $ ping 192.168.0.195
> > > PING 192.168.0.195 (192.168.0.195) 56(84) bytes of data.
> > > 64 bytes from 192.168.0.195: icmp_seq=1 ttl=64 time=0.554 ms
> > > 64 bytes from 192.168.0.195: icmp_seq=2 ttl=64 time=0.499 ms
> > > 64 bytes from 192.168.0.195: icmp_seq=3 ttl=64 time=0.502 ms
> > > 64 bytes from 192.168.0.195: icmp_seq=4 ttl=64 time=0.486 ms
> > >
> > > Performance testing using iperf revealed no noticeable impact on
> > > network throughput or CPU utilization.
> >
> > I'm not sure this short paragraph addresses Andrew's comment:
> >
> >   Have you benchmarked CPU usage with this patch, for a range of traffic
> >   bandwidths and burst patterns. How does it differ?
> >
> > Maybe you could provide more details of the iperf tests you ran? It
> > seems odd that CPU usage is unchanged.
> >
> > If the system is more reactive (due to lower coalesce settings and
> > IRQs firing more often), you'd expect CPU usage to increase,
> > wouldn't you?
> 
> Hi Joe
> 
> It is not as simple as that.
> 
> Consider a VoIP system, a CISCO or Snom phone. It will be receiving a
> packet about every 2ms. This change in interrupt coalescing will have
> no effect on CPU load, there will still be an interrupt per
> packet. What this change does however do is reduce the latency, as can
> be seen by the ping. However, anybody building a phone knows about
> 
> ethtool -C|--coalesce
> 
> and will either configure the value lower, or turn it off
> altogether. Also, CCITT recommends 50ms end to end delay for a
> national call, so going from 1.5 to 0.4ms is in the noise.
> 
> Now consider bulk transfer at line rate. The receive buffer is going
> to fill with multiple packets, NAPI is going to get its budget of 64
> packets, and the interrupt will be left disabled. NAPI will then poll
> the device every so often, receiving packets. Since interrupts are
> off, the coalesce time makes no difference.
> 
> Now consider packets arriving at about 0.5ms intervals. That is way
> too slow for NAPI to go into polled mode. It does however mean 2
> packets would typically be received in each coalescence
> period. However with the proposed change, an interrupt would be
> triggered for each packet, doubling the interrupt load.
> 
> But think about a packet every 0.5ms. That is 2000 packets per
> second. Even the older CPUs should be able to handle that.
> 
> What i would really like to know is the real use case this change is
> for. For my, ping is not a use case.
> 

Sorry for the delayed response, as I really didn't have a real use case
before. But just the other day we had a customer issue. The customer
used eth0 (FEC) and eth1 (eQOS, Synopsys DWMAC) on i.MX8MP. Both
ports are RMII with 100M bandwidth. They tested and found that the
transmission rate of both ports was different when using TFTP to transfer
data, FEC was slower than eQOS. The transmission rate of FEC was only
about 400KB, but the eQOS could reach to 1MB. When the related 
coalescence parameters were adjusted to be the same as eQOS, the
transmission rate was the same as eQOS.

From this use case, when there are fewer packets, if the coalescence
parameters are set to large values, the CPU will respond too slowly, thus
affecting the performance of some protocols.

Powered by blists - more mailing lists