linux-kernel - [PATCH net] net/mlx5: Avoid deadlock between PCI error recovery and health reporter

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <20250807131130.4056349-1-gbayer@linux.ibm.com>
Date: Thu,  7 Aug 2025 15:11:30 +0200
From: Gerd Bayer <gbayer@...ux.ibm.com>
To: Saeed Mahameed <saeedm@...dia.com>, Leon Romanovsky <leon@...nel.org>,
        Tariq Toukan <tariqt@...dia.com>, Mark Bloch <mbloch@...dia.com>,
        Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
        Moshe Shemesh <moshe@...dia.com>, Shay Drori <shayd@...dia.com>
Cc: Niklas Schnelle <schnelle@...ux.ibm.com>,
        Alexandra Winter <wintera@...ux.ibm.com>,
        Gerd Bayer <gbayer@...ux.ibm.com>, Andrew Lunn <andrew+netdev@...n.ch>,
        "David S. Miller" <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>,
        Yuanyuan Zhong <yzhong@...estorage.com>,
        Mohamed Khalfella <mkhalfella@...estorage.com>, netdev@...r.kernel.org,
        linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: [PATCH net] net/mlx5: Avoid deadlock between PCI error recovery and health reporter

During error recovery testing a pair of tasks was reported to be hung
due to a dead-lock situation:

- mlx5_unload_one() trying to acquire devlink lock while the PCI error
  recovery code had acquired the pci_cfg_access_lock().
- mlx5_crdump_collect() trying to acquire the pci_cfg_access_lock()
  while devlink_health_report() had acquired the devlink lock.

Move the check for pci_channel_offline prior to acquiring the
pci_cfg_access_lock in mlx5_vsc_gw_lock since collecting the crdump will
fail anyhow while PCI error recovery is running.

Fixes: 33afbfcc105a ("net/mlx5: Stop waiting for PCI if pci channel is offline")
Signed-off-by: Gerd Bayer <gbayer@...ux.ibm.com>
---

Hi all,

while the initial hit was recorded during "random" testing, where PCI
error recovery and poll_health() tripped almost simultaneously, I found
a way to reproduce a very similar hang at will on s390:

Inject a PCI error recovery event on a Physical Function <BDF> with
  zpcictl --reset-fw <BDF>

then request a crdump with
  devlink health dump show pci/<BDF> reporter fw_fatal

With the patch applied I didn't get the hang but kernel logs showed:
[  792.885743] mlx5_core 000a:00:00.0: mlx5_crdump_collect:51:(pid 1415): crdump: failed to lock vsc gw err -13

and the crdump request ended with:
kernel answers: Permission denied

Thanks, Gerd
---
 drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c
index 432c98f2626d..d2d3b57a57d5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c
@@ -73,16 +73,15 @@ int mlx5_vsc_gw_lock(struct mlx5_core_dev *dev)
 	u32 lock_val;
 	int ret;
 
+	if (pci_channel_offline(dev->pdev))
+		return -EACCES;
+
 	pci_cfg_access_lock(dev->pdev);
 	do {
 		if (retries > VSC_MAX_RETRIES) {
 			ret = -EBUSY;
 			goto pci_unlock;
 		}
-		if (pci_channel_offline(dev->pdev)) {
-			ret = -EACCES;
-			goto pci_unlock;
-		}
 
 		/* Check if semaphore is already locked */
 		ret = vsc_read(dev, VSC_SEMAPHORE_OFFSET, &lock_val);
-- 
2.48.1