Message-ID: <fa332bfc-68fb-4eea-a70a-8ac9c0d3c990@raritan.com>
Date: Mon, 29 Apr 2024 13:46:29 +0200
From: Ronald Wahl <ronald.wahl@...itan.com>
To: Marek Vasut <marex@...x.de>, netdev@...r.kernel.org
Cc: "David S. Miller" <davem@...emloft.net>,
 Uwe Kleine-König <u.kleine-koenig@...gutronix.de>,
 Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
 Dmitry Torokhov <dmitry.torokhov@...il.com>,
 Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Mark Brown <broonie@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
 Simon Horman <horms@...nel.org>
Subject: Re: [PATCH 2/2] net: ks8851: Handle softirqs at the end of IRQ thread
 to fix hang

Hi,

for the spi version of the chip this change now leads to

[   23.793000] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283
[   23.801915] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 857, name: irq/52-eth-link
[   23.810895] preempt_count: 200, expected: 0
[   23.815288] CPU: 0 PID: 857 Comm: irq/52-eth-link Not tainted 6.6.28-sama5 #1
[   23.822790] Hardware name: Atmel SAMA5
[   23.826717]  unwind_backtrace from show_stack+0xb/0xc
[   23.831992]  show_stack from dump_stack_lvl+0x19/0x1e
[   23.837433]  dump_stack_lvl from __might_resched+0xb7/0xec
[   23.843122]  __might_resched from mutex_lock+0xf/0x2c
[   23.848540]  mutex_lock from ks8851_irq+0x1f/0x164
[   23.853525]  ks8851_irq from irq_thread_fn+0xf/0x28
[   23.858776]  irq_thread_fn from irq_thread+0x93/0x130
[   23.864037]  irq_thread from kthread+0x7f/0x90
[   23.868699]  kthread from ret_from_fork+0x11/0x1c

Actually the spi driver variant does not suffer from the issue as it has
different locking so we probably should do the
local_bh_disable/local_bh_enable only for the "par" version. What do you think?

- ron

On 31.03.24 16:21, Marek Vasut wrote:
> [You don't often get email from marex@...x.de. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> The ks8851_irq() thread may call ks8851_rx_pkts() in case there are
> any packets in the MAC FIFO, which calls netif_rx(). This netif_rx()
> implementation is guarded by local_bh_disable() and local_bh_enable().
> The local_bh_enable() may call do_softirq() to run softirqs in case
> any are pending. One of the softirqs is net_rx_action, which ultimately
> reaches the driver .start_xmit callback. If that happens, the system
> hangs. The entire call chain is below:
>
> ks8851_start_xmit_par from netdev_start_xmit
> netdev_start_xmit from dev_hard_start_xmit
> dev_hard_start_xmit from sch_direct_xmit
> sch_direct_xmit from __dev_queue_xmit
> __dev_queue_xmit from __neigh_update
> __neigh_update from neigh_update
> neigh_update from arp_process.constprop.0
> arp_process.constprop.0 from __netif_receive_skb_one_core
> __netif_receive_skb_one_core from process_backlog
> process_backlog from __napi_poll.constprop.0
> __napi_poll.constprop.0 from net_rx_action
> net_rx_action from __do_softirq
> __do_softirq from call_with_stack
> call_with_stack from do_softirq
> do_softirq from __local_bh_enable_ip
> __local_bh_enable_ip from netif_rx
> netif_rx from ks8851_irq
> ks8851_irq from irq_thread_fn
> irq_thread_fn from irq_thread
> irq_thread from kthread
> kthread from ret_from_fork
>
> The hang happens because ks8851_irq() first locks a spinlock in
> ks8851_par.c ks8851_lock_par() spin_lock_irqsave(&ksp->lock, ...)
> and with that spinlock locked, calls netif_rx(). Once the execution
> reaches ks8851_start_xmit_par(), it calls ks8851_lock_par() again
> which attempts to claim the already locked spinlock again, and the
> hang happens.
>
> Move the do_softirq() call outside of the spinlock protected section
> of ks8851_irq() by disabling BHs around the entire spinlock protected
> section of ks8851_irq() handler. Place local_bh_enable() outside of
> the spinlock protected section, so that it can trigger do_softirq()
> without the ks8851_par.c ks8851_lock_par() spinlock being held, and
> safely call ks8851_start_xmit_par() without attempting to lock the
> already locked spinlock.
>
> Since ks8851_irq() is protected by local_bh_disable()/local_bh_enable()
> now, replace netif_rx() with __netif_rx() which is not duplicating the
> local_bh_disable()/local_bh_enable() calls.
>
> Fixes: 797047f875b5 ("net: ks8851: Implement Parallel bus operations")
> Signed-off-by: Marek Vasut <marex@...x.de>
> ---
> Cc: "David S. Miller" <davem@...emloft.net>
> Cc: "Uwe Kleine-König" <u.kleine-koenig@...gutronix.de>
> Cc: Andy Shevchenko <andriy.shevchenko@...ux.intel.com>
> Cc: Dmitry Torokhov <dmitry.torokhov@...il.com>
> Cc: Eric Dumazet <edumazet@...gle.com>
> Cc: Jakub Kicinski <kuba@...nel.org>
> Cc: Mark Brown <broonie@...nel.org>
> Cc: Paolo Abeni <pabeni@...hat.com>
> Cc: Ronald Wahl <ronald.wahl@...itan.com>
> Cc: Simon Horman <horms@...nel.org>
> Cc: netdev@...r.kernel.org
> ---
>   drivers/net/ethernet/micrel/ks8851_common.c | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/micrel/ks8851_common.c b/drivers/net/ethernet/micrel/ks8851_common.c
> index 896d43bb8883d..b6b727e651f3d 100644
> --- a/drivers/net/ethernet/micrel/ks8851_common.c
> +++ b/drivers/net/ethernet/micrel/ks8851_common.c
> @@ -299,7 +299,7 @@ static void ks8851_rx_pkts(struct ks8851_net *ks)
>                                          ks8851_dbg_dumpkkt(ks, rxpkt);
>
>                                  skb->protocol = eth_type_trans(skb, ks->netdev);
> -                               netif_rx(skb);
> +                               __netif_rx(skb);
>
>                                  ks->netdev->stats.rx_packets++;
>                                  ks->netdev->stats.rx_bytes += rxlen;
> @@ -325,11 +325,15 @@ static void ks8851_rx_pkts(struct ks8851_net *ks)
>    */
>   static irqreturn_t ks8851_irq(int irq, void *_ks)
>   {
> +       bool need_bh_off = !(hardirq_count() | softirq_count());
>          struct ks8851_net *ks = _ks;
>          unsigned handled = 0;
>          unsigned long flags;
>          unsigned int status;
>
> +       if (need_bh_off)
> +               local_bh_disable();
> +
>          ks8851_lock(ks, &flags);
>
>          status = ks8851_rdreg16(ks, KS_ISR);
> @@ -406,6 +410,9 @@ static irqreturn_t ks8851_irq(int irq, void *_ks)
>          if (status & IRQ_LCI)
>                  mii_check_link(&ks->mii);
>
> +       if (need_bh_off)
> +               local_bh_enable();
> +
>          return IRQ_HANDLED;
>   }
>
> --
> 2.43.0
>


________________________________

Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

Powered by blists - more mailing lists