Date: Mon, 24 Jun 2024 10:25:38 +0100
From: "Lad, Prabhakar" <prabhakar.csengg@...il.com>
To: Alexandre Torgue <alexandre.torgue@...s.st.com>, Jose Abreu <joabreu@...opsys.com>
Cc: netdev <netdev@...r.kernel.org>
Subject: STMMAC driver CPU stall warning

Hi Alexandre and Jose,

While integrating the STMMAC driver for the Renesas RZ/V2H SoC I am
seeing the CPU stall warning when the locking config options are
enabled. Has anyone seen this issue?

Platform and Kernel details :
-------------------------------------
- Renesas RZ/V2H(P) EVK (ARM64)
- 6.10.0-rc5
- IP version, Synopsys ID: 0x52

With the below locking configs enabled:
----------------------------------------------------
# CONFIG_PER_VMA_LOCK_STATS is not set
# CONFIG_SOFTLOCKUP_DETECTOR is not set
CONFIG_HAVE_HARDLOCKUP_DETECTOR_BUDDY=y
# CONFIG_HARDLOCKUP_DETECTOR is not set
CONFIG_LOCK_DEBUGGING_SUPPORT=y
CONFIG_PROVE_LOCKING=y
# CONFIG_PROVE_RAW_LOCK_NESTING is not set
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_LOCKDEP=y
CONFIG_LOCKDEP_BITS=15
CONFIG_LOCKDEP_CHAINS_BITS=16
CONFIG_LOCKDEP_STACK_TRACE_BITS=19
CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
# CONFIG_DEBUG_LOCKDEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_CSD_LOCK_WAIT_DEBUG is not set
CONFIG_TRACE_CLOCK=y

[   33.701954] rcu: INFO: rcu_preempt self-detected stall on CPU
[   33.707776] rcu:     0-....: (1 GPs behind)
idle=7054/1/0x4000000000000000 softirq=413/414 fqs=2228
[   33.716693] rcu:     (t=6500 jiffies g=-691 q=65 ncpus=4)
[   33.721856] CPU: 0 PID: 37 Comm: kworker/u16:1 Not tainted
6.10.0-rc5-arm64-renesas-00383-g34de6e9c5801 #495
[   33.731717] Hardware name: Renesas based on r9a09g057h44 (DT)
[   33.737833] Workqueue: rpciod rpc_async_schedule
[   33.742488] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   33.749474] pc : _raw_spin_unlock_irqrestore+0x34/0x68
[   33.754640] lr : _raw_spin_unlock_irqrestore+0x64/0x68
[   33.759800] sp : ffff8000828133b0
[   33.763128] x29: ffff8000828133b0 x28: 0000000000000080 x27: 0000000000000000
[   33.770304] x26: ffff8000818faa00 x25: ffff0003ee18b200 x24: 0000000000000001
[   33.777478] x23: ffff0003ee18b180 x22: ffff0003ee18b180 x21: ffff0003ee18b180
[   33.784653] x20: ffff0003ee18b180 x19: 0000000000000000 x18: 00000000000020e8
[   33.791827] x17: 344617ab1e4d0a08 x16: 0101000003d71d0f x15: 0000000000000028
[   33.799002] x14: 000000000000000a x13: 0000000000004d7c x12: 0000000000000000
[   33.806176] x11: ffff800082540a7c x10: ffff8000814e3cd4 x9 : 0000000000000168
[   33.813351] x8 : 0000000000000001 x7 : 00000000000000c8 x6 : ffff8000821e6000
[   33.820524] x5 : 0000000000000000 x4 : 0000000000000009 x3 : 0000000000000000
[   33.827698] x2 : 0000000000000001 x1 : ffff8000814e3cd0 x0 : 0000000000009384
[   33.834872] Call trace:
[   33.837331]  _raw_spin_unlock_irqrestore+0x34/0x68
[   33.842145]  hrtimer_start_range_ns+0x138/0x314
[   33.846700]  stmmac_tx_timer_arm+0x88/0x8c
[   33.850819]  stmmac_xmit+0x778/0xac4
[   33.854413]  dev_hard_start_xmit+0xcc/0x2d0
[   33.858618]  sch_direct_xmit+0x88/0x378
[   33.862477]  __dev_queue_xmit+0x784/0xfc0
[   33.866506]  ip_finish_output2+0x794/0xac8
[   33.870627]  __ip_finish_output+0x110/0x27c
[   33.874831]  ip_output+0x6c/0x278
[   33.878165]  __ip_queue_xmit+0x194/0x5a0
[   33.882106]  ip_queue_xmit+0x10/0x18
[   33.885698]  __tcp_transmit_skb+0x4a8/0xba4
[   33.889903]  __tcp_send_ack.part.0+0xc8/0x144
[   33.894280]  tcp_release_cb+0x1ec/0x208
[   33.898133]  release_sock+0x48/0x9c
[   33.901642]  tcp_sendmsg+0x44/0x58
[   33.905063]  inet_sendmsg+0x40/0x64
[   33.908575]  sock_sendmsg+0x7c/0xe0
[   33.912085]  xprt_sock_sendmsg+0xdc/0x300
[   33.916115]  xs_tcp_send_request+0xec/0x240
[   33.920318]  xprt_transmit+0x1e4/0x5e8
[   33.924084]  call_transmit+0x7c/0x90
[   33.927680]  __rpc_execute+0xdc/0x6f8
[   33.931361]  rpc_async_schedule+0x28/0x40
[   33.935390]  process_one_work+0x21c/0x620
[   33.939422]  worker_thread+0x1a8/0x374
[   33.943190]  kthread+0x10c/0x110
[   33.946436]  ret_from_fork+0x10/0x20
[   33.950033] Sending NMI from CPU 0 to CPUs 3:
[   33.954415] NMI backtrace for cpu 3
[   33.957929] CPU: 3 PID: 67 Comm: kworker/u16:6 Not tainted
6.10.0-rc5-arm64-renesas-00383-g34de6e9c5801 #495
[   33.967788] Hardware name: Renesas based on r9a09g057h44 (DT)
[   33.973902] Workqueue: xprtiod xs_stream_data_receive_workfn
[   33.979591] pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   33.986576] pc : queued_spin_lock_slowpath+0x68/0x514
[   33.991650] lr : do_raw_spin_lock+0xb8/0x104
[   33.995941] sp : ffff800082fcba70
[   33.999268] x29: ffff800082fcba70 x28: 0000000000000000 x27: 0000000000000000
[   34.006441] x26: ffff0000c2966840 x25: 0000000000000000 x24: ffff800081863000
[   34.013614] x23: ffff0000c0578000 x22: 000000000000000c x21: 0000000000000040
[   34.020786] x20: ffff8000814e1008 x19: ffff0000c29f0190 x18: 0000000000093930
[   34.027958] x17: 0019001a00400009 x16: 0038004000000000 x15: 0000000000000000
[   34.035130] x14: 0000000000000001 x13: ffff0000c2a20080 x12: 0000000000000000
[   34.042302] x11: ffff800082540a7c x10: ffff8000814e3cd4 x9 : 00000000000000a0
[   34.049473] x8 : 0000000000000001 x7 : 00000000000000c8 x6 : ffff8000821e6000
[   34.056645] x5 : f647258a66f2c9bc x4 : ffff8000821ad9e0 x3 : 0000000000000000
[   34.063815] x2 : ffff80036cd00000 x1 : 0000000000000000 x0 : 0000000000000001
[   34.070987] Call trace:
[   34.073444]  queued_spin_lock_slowpath+0x68/0x514
[   34.078169]  do_raw_spin_lock+0xb8/0x104
[   34.082110]  _raw_spin_lock_bh+0x54/0x60
[   34.086051]  lock_sock_nested+0x3c/0x84
[   34.089905]  tcp_recvmsg+0x5c/0x1c0
[   34.093412]  inet_recvmsg+0x60/0x244
[   34.097007]  sock_recvmsg+0x2c/0x6c
[   34.100515]  xs_read_stream.constprop.0+0xb0/0x4e0
[   34.105327]  xs_stream_data_receive_workfn+0x54/0x1b4
[   34.110398]  process_one_work+0x21c/0x620
[   34.114427]  worker_thread+0x1a8/0x374
[   34.118194]  kthread+0x10c/0x110
[   34.121438]  ret_from_fork+0x10/0x20
[   68.470538] random: crng init done

Below is the dmesg logs for gbeth:
----------------------------------------------
[    1.611307] renesas-gbeth 15c30000.ethernet: IRQ sfty not found
[    1.626475] renesas-gbeth 15c30000.ethernet: User ID: 0x1, Synopsys ID: 0x52
[    1.633615] renesas-gbeth 15c30000.ethernet:         DWMAC4/5
[    1.638809] renesas-gbeth 15c30000.ethernet: DMA HW capability
register supported
[    1.646351] renesas-gbeth 15c30000.ethernet: RX Checksum Offload
Engine supported
[    1.653904] renesas-gbeth 15c30000.ethernet: Wake-Up On Lan supported
[    1.660516] renesas-gbeth 15c30000.ethernet: Enable RX Mitigation
via HW Watchdog Timer
[    1.668584] renesas-gbeth 15c30000.ethernet: Enabled L3L4 Flow TC (entries=8)
[    1.675782] renesas-gbeth 15c30000.ethernet: Enabled RFS Flow TC (entries=10)
[    1.682979] renesas-gbeth 15c30000.ethernet: Using 32/32 bits DMA
host/device width
[    1.698226] renesas-gbeth 15c40000.ethernet: IRQ sfty not found
[    1.713295] renesas-gbeth 15c40000.ethernet: User ID: 0x0, Synopsys ID: 0x52
[    1.720432] renesas-gbeth 15c40000.ethernet:         DWMAC4/5
[    1.725637] renesas-gbeth 15c40000.ethernet: DMA HW capability
register supported
[    1.733179] renesas-gbeth 15c40000.ethernet: RX Checksum Offload
Engine supported
[    1.740720] renesas-gbeth 15c40000.ethernet: Wake-Up On Lan supported
[    1.747327] renesas-gbeth 15c40000.ethernet: Enable RX Mitigation
via HW Watchdog Timer
[    1.755408] renesas-gbeth 15c40000.ethernet: device MAC address
ba:91:02:27:30:8f
[    1.762977] renesas-gbeth 15c40000.ethernet: Enabled L3L4 Flow TC (entries=8)
[    1.770185] renesas-gbeth 15c40000.ethernet: Enabled RFS Flow TC (entries=10)
[    1.777382] renesas-gbeth 15c40000.ethernet: Using 32/32 bits DMA
host/device width
[    2.240115] renesas-gbeth 15c30000.ethernet eth0: Register
MEM_TYPE_PAGE_POOL RxQ-0
[    2.248621] renesas-gbeth 15c30000.ethernet eth0: Register
MEM_TYPE_PAGE_POOL RxQ-1
[    2.257069] renesas-gbeth 15c30000.ethernet eth0: Register
MEM_TYPE_PAGE_POOL RxQ-2
[    2.265151] renesas-gbeth 15c30000.ethernet eth0: Register
MEM_TYPE_PAGE_POOL RxQ-3
[    2.341611] renesas-gbeth 15c30000.ethernet eth0: PHY [stmmac-0:00]
driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
[    2.357928] renesas-gbeth 15c30000.ethernet eth0: No Safety
Features support found
[    2.365581] renesas-gbeth 15c30000.ethernet eth0: IEEE 1588-2008
Advanced Timestamp supported
[    2.375086] renesas-gbeth 15c30000.ethernet eth0: registered PTP clock
[    2.382575] renesas-gbeth 15c30000.ethernet eth0: configuring for
phy/rgmii-id link mode
[    2.400349] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-0
[    2.409133] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-1
[    2.417244] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-2
[    2.425629] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-3
[    2.466041] renesas-gbeth 15c40000.ethernet eth1: PHY [stmmac-1:00]
driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
[    2.482310] renesas-gbeth 15c40000.ethernet eth1: No Safety
Features support found
[    2.489977] renesas-gbeth 15c40000.ethernet eth1: IEEE 1588-2008
Advanced Timestamp supported
[    2.499184] renesas-gbeth 15c40000.ethernet eth1: registered PTP clock
[    2.506662] renesas-gbeth 15c40000.ethernet eth1: configuring for
phy/rgmii-id link mode
[    6.498736] renesas-gbeth 15c30000.ethernet eth0: Link is Up -
1Gbps/Full - flow control rx/tx
[   73.941601] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-0
[   73.954665] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-1
[   73.963942] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-2
[   73.976780] renesas-gbeth 15c40000.ethernet eth1: Register
MEM_TYPE_PAGE_POOL RxQ-3
[   74.066033] renesas-gbeth 15c40000.ethernet eth1: PHY [stmmac-1:00]
driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
[   74.086068] renesas-gbeth 15c40000.ethernet eth1: No Safety
Features support found
[   74.098223] renesas-gbeth 15c40000.ethernet eth1: IEEE 1588-2008
Advanced Timestamp supported
[   74.102485] renesas-gbeth 15c40000.ethernet eth1: registered PTP clock
[   74.122445] renesas-gbeth 15c40000.ethernet eth1: configuring for
phy/rgmii-id link mode

Please let me know if you need me to try anything or provide more
debug information.

Cheers,
Prabhakar

Powered by blists - more mailing lists