Date:   Thu, 30 Mar 2023 15:58:15 +0200
From:   Frank Wunderlich <frank-w@...lic-files.de>
To:     Felix Fietkau <nbd@....name>
Cc:     netdev@...r.kernel.org, Daniel Golle <daniel@...rotopia.org>
Subject: Aw: Re:  Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx
 throughput regression with direct 1G links

something ist still strange...i get a rcu stall again with this patch...reverted it and my r2 boots again.

[   29.772755] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   29.778689] rcu:     2-...0: (1 GPs behind) idle=547c/1/0x40000000 softirq=251/258 fqs=427
[   29.786697] rcu:     (detected by 1, t=2104 jiffies, g=-875, q=29 ncpus=4)
[   29.793308] Sending NMI from CPU 1 to CPUs 2:
[   34.492968] vusb: disabling
[   34.495828] vmc: disabling
[   34.498587] vmch: disabling
[   34.501433] vgp1: disabling
[   34.504426] vcamaf: disabling
[   39.797579] rcu: rcu_sched kthread timer wakeup didn't happen for 994 jiffies! g-875 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[   39.808619] rcu:     Possible timer handling issue on cpu=1 timer-softirq=493
[   39.815487] rcu: rcu_sched kthread starved for 995 jiffies! g-875 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
[   39.825571] rcu:     Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[   39.834520] rcu: RCU grace-period kthread stack dump:
[   39.839564] task:rcu_sched       state:I stack:0     pid:14    ppid:2      flags:0x00000000
[   39.847928]  __schedule from schedule+0x54/0xe8
[   39.852472]  schedule from schedule_timeout+0x94/0x158
[   39.857619]  schedule_timeout from rcu_gp_fqs_loop+0x12c/0x50c
[   39.863467]  rcu_gp_fqs_loop from rcu_gp_kthread+0x194/0x21c
[   39.869135]  rcu_gp_kthread from kthread+0xc8/0xcc
[   39.873931]  kthread from ret_from_fork+0x14/0x2c
[   39.878639] Exception stack(0xf0859fb0 to 0xf0859ff8)
[   39.883690] 9fa0:                                     00000000 00000000 00000000 00000000
[   39.891864] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   39.900037] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   39.906645] rcu: Stack dump where RCU GP kthread last ran:
[   39.912125] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-rc1-bpi-r2-rc-net #2
[   39.919518] Hardware name: Mediatek Cortex-A7 (Device Tree)
[   39.925082] PC is at default_idle_call+0x1c/0xb0
[   39.929698] LR is at ct_kernel_enter.constprop.0+0x48/0x11c
[   39.935267] pc : [<c0d105ec>]    lr : [<c0d0ffa4>]    psr: 600e0013
[   39.941527] sp : f0861fb0  ip : c15721e0  fp : 00000000
[   39.946746] r10: 00000000  r9 : 410fc073  r8 : 8000406a
[   39.951964] r7 : c1404f74  r6 : c19e0900  r5 : c15727e0  r4 : c19e0900
[   39.958486] r3 : 00000000  r2 : 2da0a000  r1 : 00000001  r0 : 00008cfc
[   39.965007] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   39.972138] Control: 10c5387d  Table: 84f4806a  DAC: 00000051
[   39.977878]  default_idle_call from cpuidle_idle_call+0x24/0x68
[   39.983805]  cpuidle_idle_call from do_idle+0x9c/0xd0
[   39.988863]  do_idle from cpu_startup_entry+0x20/0x24
[   39.993921]  cpu_startup_entry from secondary_start_kernel+0x118/0x138
[   40.000457]  secondary_start_kernel from 0x801017a0

maybe i need additional patch or did anything else wrong?

still working on 6.3-rc1
https://github.com/frank-w/BPI-Router-Linux/commits/6.3-rc-net

regards Frank


> Gesendet: Mittwoch, 29. März 2023 um 14:04 Uhr
> Von: "Felix Fietkau" <nbd@....name>
> An: "Frank Wunderlich" <frank-w@...lic-files.de>
> Cc: netdev@...r.kernel.org, "Daniel Golle" <daniel@...rotopia.org>
> Betreff: Re: Aw: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>
> On 27.03.23 19:28, Frank Wunderlich wrote:
> > 
> >> Gesendet: Sonntag, 26. März 2023 um 22:09 Uhr
> >> Von: "Felix Fietkau" <nbd@....name>
> >> On 26.03.23 19:49, Frank Wunderlich wrote:
> >> >> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
> >> >> Von: "Felix Fietkau" <nbd@....name>
> > 
> >> >> On 26.03.23 19:10, Frank Wunderlich wrote:
> >> >> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
> >> >> >> Von: "Felix Fietkau" <nbd@....name>
> > 
> >> >> >> On 25.03.23 10:28, Frank Wunderlich wrote:
> >> >> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
> >> >> >> >> Von: "Felix Fietkau" <nbd@....name>
> > 
> >> >> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
> >> >> >> > 
> >> >> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
> >> >> >> > 
> >> >> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
> >> >> >> Could you please test if making it use the MT7621 codepath brings back 
> >> >> >> performance? I don't have any MT7623 hardware for testing right now.
> > 
> >> > used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue
> >> > 
> >> > and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)
> >> Does the MAC that's connected to the switch use flow control? Can you 
> >> test if changing that makes a difference?
> > 
> > it does use flow control/pause on mac and switch-port, disabled it, but it does not change anything (still ~620Mbit on tx)
> 
> I finally found a MT7623 device in my stash, so I could run some
> experiments with it. For some reason I could only reproduce your tx
> throughput values after switching off TRGMII. With TRGMII enabled, I got
> around 864 Mbit/s, which is of course still lower than what I get with
> shaping disabled.
> I also experimented with bumping the shaping rate to higher values, but
> got no changes in throughput at all.
> Based on that, I'm beginning to think that the shaper simply can't
> handle rates close to the MAC rate and runs into a fundamental limit
> somehow.
> Now that I think about it, I do remember that shaping to 2.5 Gbps on
> MT7622 also reduced link throughput.
> 
> I think we should simply use this patch to deal with it. Do you agree?
> 
> - Felix
> 
> ---
> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> @@ -753,6 +753,7 @@ static void mtk_mac_link_up(struct phylink_config *config,
>   		 MAC_MCR_FORCE_RX_FC);
>   
>   	/* Configure speed */
> +	mac->speed = speed;
>   	switch (speed) {
>   	case SPEED_2500:
>   	case SPEED_1000:
> @@ -3235,6 +3236,9 @@ static int mtk_device_event(struct notifier_block *n, unsigned long event, void
>   	if (dp->index >= MTK_QDMA_NUM_QUEUES)
>   		return NOTIFY_DONE;
>   
> +	if (mac->speed > 0 && mac->speed <= s.base.speed)
> +		s.base.speed = 0;
> +
>   	mtk_set_queue_speed(eth, dp->index + 3, s.base.speed);
>   
>   	return NOTIFY_DONE;
> 
> 
> 
>

Powered by blists - more mailing lists