| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAODvEq4HK0isNpT5G-O3zKAgThiLt39hz_h8eQS4-Vnf1NjJnw@mail.gmail.com>
Date: Wed, 7 Jan 2026 10:39:59 -0800
From: Li Li <boolli@...gle.com>
To: Paul Menzel <pmenzel@...gen.mpg.de>
Cc: Tony Nguyen <anthony.l.nguyen@...el.com>,
Przemek Kitszel <przemyslaw.kitszel@...el.com>, "David S. Miller" <davem@...emloft.net>,
Jakub Kicinski <kuba@...nel.org>, Eric Dumazet <edumazet@...gle.com>, intel-wired-lan@...ts.osuosl.org,
netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
David Decotigny <decot@...gle.com>, Anjali Singhai <anjali.singhai@...el.com>,
Sridhar Samudrala <sridhar.samudrala@...el.com>, Brian Vazquez <brianvv@...gle.com>,
emil.s.tantilov@...el.com
Subject: Re: [Intel-wired-lan] [PATCH 1/5] idpf: skip getting/setting ring
params if vport is NULL during HW reset
On Tue, Jan 6, 2026 at 9:30 PM Paul Menzel <pmenzel@...gen.mpg.de> wrote:
>
> Dear Li,
>
>
> Thank you for your patch.
>
> Am 07.01.26 um 02:04 schrieb Li Li via Intel-wired-lan:
> > When an idpf HW reset is triggered, it clears the vport but does
> > not clear the netdev held by vport:
> >
> > // In idpf_vport_dealloc() called by idpf_init_hard_reset(),
> > // idpf_init_hard_reset() sets IDPF_HR_RESET_IN_PROG, so
> > // idpf_decfg_netdev() doesn't get called.
>
> No need to format this as code comments. At least it confused me a little.
Thanks for the pointer. Will drop the comment format in the future.
>
> > if (!test_bit(IDPF_HR_RESET_IN_PROG, adapter->flags))
> > idpf_decfg_netdev(vport);
> > // idpf_decfg_netdev() would clear netdev but it isn't called:
> > unregister_netdev(vport->netdev);
> > free_netdev(vport->netdev);
> > vport->netdev = NULL;
> > // Later in idpf_init_hard_reset(), the vport is cleared:
> > kfree(adapter->vports);
> > adapter->vports = NULL;
> >
> > During an idpf HW reset, when "ethtool -g/-G" is called on the netdev,
> > the vport associated with the netdev is NULL, and so a kernel panic
> > would happen:
> >
> > [ 513.185327] BUG: kernel NULL pointer dereference, address: 0000000000000038
> > ...
> > [ 513.232756] RIP: 0010:idpf_get_ringparam+0x45/0x80
> >
> > This can be reproduced reliably by injecting a TX timeout to cause
> > an idpf HW reset, and injecting a virtchnl error to cause the HW
> > reset to fail and retry, while calling "ethtool -g/-G" on the netdev
> > at the same time.
>
> If you shared the commands, how to do that, it would make reproducing
> the issue easier.
Here's what I did to introduce TX timeouts and virtchnl timeouts at run time:
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
@@ -15,6 +15,9 @@ struct idpf_tx_stash {
#define idpf_tx_buf_compl_tag(buf) (*(u32 *)&(buf)->priv)
LIBETH_SQE_CHECK_PRIV(u32);
+static bool SIMULATE_TX_TIMEOUT;
+module_param(SIMULATE_TX_TIMEOUT, bool, 0644);
+
/**
* idpf_chk_linearize - Check if skb exceeds max descriptors per packet
* @skb: send buffer
@@ -79,6 +82,8 @@ void idpf_tx_timeout(struct net_device *netdev,
unsigned int txqueue)
adapter->tx_timeout_count++;
+ SIMULATE_TX_TIMEOUT = false;
+
netdev_err(netdev, "Detected Tx timeout: Count %d, Queue %d\n",
adapter->tx_timeout_count, txqueue);
if (!idpf_is_reset_in_prog(adapter)) {
@@ -2028,6 +2033,12 @@ static bool idpf_tx_clean_complq(struct
idpf_compl_queue *complq, int budget,
}
tx_q = complq->txq_grp->txqs[rel_tx_qid];
+ if (unlikely(SIMULATE_TX_TIMEOUT && (tx_q->idx == 1))) {
+ netdev_err(tx_q->netdev, "boolli test:
triggering TX timeout for TX queue id %d\n", tx_q->idx);
+ goto fetch_next_desc;
+ }
+
+
/* Determine completion type */
ctype = le16_get_bits(tx_desc->qid_comptype_gen,
IDPF_TXD_COMPLQ_COMPL_TYPE_M);
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
@@ -6,6 +6,9 @@
#include "idpf.h"
#include "idpf_virtchnl.h"
+static bool SIMULATE_VC_TIMEOUT;
+module_param(SIMULATE_VC_TIMEOUT, bool, 0644);
+
#define IDPF_VC_XN_MIN_TIMEOUT_MSEC 2000
#define IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC (60 * 1000)
#define IDPF_VC_XN_IDX_M GENMASK(7, 0)
@@ -800,6 +803,10 @@ static int idpf_send_ver_msg(struct idpf_adapter *adapter)
xn_params.timeout_ms = IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC;
reply_sz = idpf_vc_xn_exec(adapter, &xn_params);
+ if (SIMULATE_VC_TIMEOUT) {
+ dev_err(&adapter->pdev->dev, "boolli test: simulating
VC timeout by returning -ETIME in idpf_send_ver_msg");
+ reply_sz = -ETIME;
+ }
if (reply_sz < 0)
return reply_sz;
if (reply_sz < sizeof(vvi))
Then after the kernel is booted, we can introduce the TX timeout that
lasts forever by doing the following:
echo 1 | tee /sys/module/idpf/parameters/SIMULATE_TX_TIMEOUT && echo 1
| tee /sys/module/idpf/parameters/SIMULATE_VC_TIMEOUT
All my experiments in this patch series were performed after the
kernel was put in such a state.
>
> > With this patch applied, we see the following error but no kernel
> > panics anymore:
> >
> > [ 476.323630] idpf 0000:05:00.0 eth1: failed to get ring params due to no vport in netdev
> >
> > Signed-off-by: Li Li <boolli@...gle.com>
> > ---
> > drivers/net/ethernet/intel/idpf/idpf_ethtool.c | 12 ++++++++++++
> > 1 file changed, 12 insertions(+)
> >
> > diff --git a/drivers/net/ethernet/intel/idpf/idpf_ethtool.c b/drivers/net/ethernet/intel/idpf/idpf_ethtool.c
> > index d5711be0b8e69..6a4b630b786c2 100644
> > --- a/drivers/net/ethernet/intel/idpf/idpf_ethtool.c
> > +++ b/drivers/net/ethernet/intel/idpf/idpf_ethtool.c
> > @@ -639,6 +638,10 @@ static void idpf_get_ringparam(struct net_device *netdev,
> >
> > idpf_vport_ctrl_lock(netdev);
> > vport = idpf_netdev_to_vport(netdev);
> > + if (!vport) {
> > + netdev_err(netdev, "failed to get ring params due to no vport in netdev\n");
>
> If vport == NULL is expected, why log it as an error. What should the
> user do? Wait until reset is done?
>
> > + goto unlock;
> > + }
> >
> > ring->rx_max_pending = IDPF_MAX_RXQ_DESC;
> > ring->tx_max_pending = IDPF_MAX_TXQ_DESC;
> > @@ -647,6 +651,7 @@ static void idpf_get_ringparam(struct net_device *netdev,
> >
> > kring->tcp_data_split = idpf_vport_get_hsplit(vport);
> >
> > +unlock:
> > idpf_vport_ctrl_unlock(netdev);
> > }
> >
> > @@ -673,6 +674,11 @@ static int idpf_set_ringparam(struct net_device *netdev,
> >
> > idpf_vport_ctrl_lock(netdev);
> > vport = idpf_netdev_to_vport(netdev);
> > + if (!vport) {
> > + netdev_err(netdev, "ring params not changed due to no vport in netdev\n");
> > + err = -EFAULT;
> > + goto unlock_mutex;
> > + }
> >
> > idx = vport->idx;
> >
>
> Is there another – possible more involved – solution possible to wait
> until the hardware reset finished?
Please see Emil's patch series at
https://lore.kernel.org/intel-wired-lan/20251121001218.4565-1-emil.s.tantilov@intel.com/,
in which https://lore.kernel.org/intel-wired-lan/20251121001218.4565-3-emil.s.tantilov@intel.com/
detaches the netdev at the start of a HW reset, which I also think is
a more elegant solution than mine.
I'm going to drop this patch series in favor of Emil's solution above.
>
>
> Kind regards,
>
> Paul
Powered by blists - more mailing lists