netdev - mlx5 driver is broken when pci_msix_can_alloc

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date: Tue, 30 May 2023 15:04:36 +0200
From: Niklas Schnelle <schnelle@...ux.ibm.com>
To: Shay Drory <shayd@...dia.com>, Saeed Mahameed <saeedm@...dia.com>,
        Eli
	Cohen <elic@...dia.com>, netdev@...r.kernel.org
Subject: mlx5 driver is broken when pci_msix_can_alloc_dyn() is false with
 v6.4-rc4

Hi Saeed, Eli, Shay,

With v6.4-rc4 I'm getting a stream of RX and TX timeouts when trying to
use ConnectX-4 and ConnectX-6 VFs on s390. I've bisected this and found
the following commit to be broken:

commit 1da438c0ae02396dc5018b63237492cb5908608d
Author: Shay Drory <shayd@...dia.com>
Date:   Mon Apr 17 10:57:50 2023 +0300

    net/mlx5: Fix indexing of mlx5_irq

    After the cited patch, mlx5_irq xarray index can be different then
    mlx5_irq MSIX table index.
    Fix it by storing both mlx5_irq xarray index and MSIX table index.

    Fixes: 3354822cde5a ("net/mlx5: Use dynamic msix vectors allocation")
    Signed-off-by: Shay Drory <shayd@...dia.com>
    Reviewed-by: Eli Cohen <elic@...dia.com>
    Signed-off-by: Saeed Mahameed <saeedm@...dia.com>

The problem is that our IRQs currently still use a legacy mode instead
of a full fledged IRQ domain. One consequence of that is that
pci_msix_can_alloc_dyn(dev->pdev) returns false. That lands us in the
non dynamic case in mlx5_irq_alloc() where irq->map.index is set to 0.
Now prior to the above commit irq->map.index would later be set to i
(the irq number) but that was replaced with just setting irq-
>pool_index = i. For the dynamic case this is fine because
pci_msix_alloc_irq_at() sets it but for the non-dynamic case this leave
it unset. With the following diff the RX/TX timeouts go away:

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
index db5687d9fec9..94dce3735204 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
@@ -237,7 +237,7 @@ struct mlx5_irq *mlx5_irq_alloc(struct
mlx5_irq_pool *pool, int i,
                 * vectors have also been allocated.
                 */
                irq->map.virq = pci_irq_vector(dev->pdev, i);
-               irq->map.index = 0;
+               irq->map.index = i;
        } else {
                irq->map = pci_msix_alloc_irq_at(dev->pdev,
MSI_ANY_INDEX, af_desc);
                if (!irq->map.virq) {



I'll sent a patch with the above shortly but wanted to give you a heads
up since I'd really like to get this fixed for -rc5 or at least -rc6.

Thanks,
Niklas