netdev - Re: [?bug] Can't get switchdev mode work on ConnectX-4 Card

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <091ec8be-34f6-4324-9e79-d2fbc102fd6b@nvidia.com>
Date: Tue, 22 Apr 2025 09:42:26 +0300
From: Mark Bloch <mbloch@...dia.com>
To: Qiyu Yan <yanqiyu17@...ls.ucas.ac.cn>, Saeed Mahameed
 <saeedm@...dia.com>, Tariq Toukan <tariqt@...dia.com>
Cc: netdev@...r.kernel.org, Ivan Vecera <ivecera@...hat.com>,
 Jiri Pirko <jiri@...nulli.us>
Subject: Re: [?bug] Can't get switchdev mode work on ConnectX-4 Card



On 21/04/2025 12:07, Qiyu Yan wrote:
> Hi,
> 
> I have a ConnectX-4 Lx EN MCX4121A-acat card:
> 
> $ lspci -s c1:00.0
> c1:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
> $ devlink dev info pci/0000:c1:00.0
> pci/0000:c1:00.0:
>   driver mlx5_core
>   versions:
>       fixed:
>         fw.psid MT_2420110034
>       running:
>         fw.version 14.32.1900
>         fw 14.32.1900
>       stored:
>         fw.version 14.32.1900
>         fw 14.32.1900
> 
> I wanted to put the card to switchdev mode, so I started trying to to the following:
> 
> # enable switchdev mode
> $ sudo devlink dev eswitch set pci/0000:c1:00.0 mode switchdev
> $ sudo devlink dev eswitch show pci/0000:c1:00.0
> pci/0000:c1:00.0: mode switchdev inline-mode link encap-mode basic
> 
> # create 2 VFs
> $ echo 2 | sudo tee /sys/class/net/mlx-p0/device/sriov_numvfs
> 
> # Try add interface to bridges
> $ sudo ip link add vmbr type bridge
> $ sudo ip link set mlx-p0 master vmbr
> Error: mlx5_core: Error checking for existing bridge with same ifindex.
> $ sudo ip link set enp193s0f0r0 master vmbr
> Error: mlx5_core: Error checking for existing bridge with same ifindex.

It’s likely that the issue stems from cx4-lx not supporting metadata
matching, which in turn prevents the driver from enabling bridge
offloads.

Could you please confirm this by checking the output of the following
command?
# devlink dev param show pci/0000:c1:00.0 name esw_port_metadata

A better approach might be to check for metadata matching support
ahead of time and avoid registering for bridge offloads if it's not
supported. This way the driver won't offload the bridge but
it will also won't prevent users from adding the reps to bridge.
Could you try the diff below and let me know if it resolves
the issue for you?

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c
index 0f5d7ea8956f..25a5845e5618 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c
@@ -523,6 +523,11 @@ void mlx5e_rep_bridge_init(struct mlx5e_priv *priv)
                mdev->priv.eswitch;
        int err;

+       if (!mlx5_esw_bridge_supported(esw)) {
+               esw_debug(mdev, "Bridge offlaods isn't supported\n");
+               return;
+       }
+
        rtnl_lock();
        br_offloads = mlx5_esw_bridge_init(esw);
        rtnl_unlock();
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c
index 76e35c827da0..37781f9ca884 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c
@@ -368,9 +368,6 @@ mlx5_esw_bridge_ingress_table_init(struct mlx5_esw_bridge_offloads *br_offloads)
        struct mlx5_eswitch *esw = br_offloads->esw;
        int err;

-       if (!mlx5_eswitch_vport_match_metadata_enabled(esw))
-               return -EOPNOTSUPP;
-
        ingress_ft = mlx5_esw_bridge_table_create(MLX5_ESW_BRIDGE_INGRESS_TABLE_SIZE,
                                                  MLX5_ESW_BRIDGE_LEVEL_INGRESS_TABLE,
                                                  esw);
@@ -1917,6 +1914,14 @@ static void mlx5_esw_bridge_flush(struct mlx5_esw_bridge_offloads *br_offloads)
                  "Cleaning up bridge offloads while still having bridges attached\n");
 }

+bool mlx5_esw_bridge_supported(struct mlx5_eswitch *esw)
+{
+       if (!mlx5_eswitch_vport_match_metadata_enabled(esw))
+               return false;
+
+       return true;
+}
+
 struct mlx5_esw_bridge_offloads *mlx5_esw_bridge_init(struct mlx5_eswitch *esw)
 {
        struct mlx5_esw_bridge_offloads *br_offloads;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h
index d6f539161993..f920c1c47f47 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h
@@ -93,5 +93,6 @@ int mlx5_esw_bridge_port_mdb_add(struct net_device *dev, u16 vport_num, u16 esw_
 void mlx5_esw_bridge_port_mdb_del(struct net_device *dev, u16 vport_num, u16 esw_owner_vhca_id,
                                  const unsigned char *addr, u16 vid,
                                  struct mlx5_esw_bridge_offloads *br_offloads);
+bool mlx5_esw_bridge_supported(struct mlx5_eswitch *esw);

 #endif /* __MLX5_ESW_BRIDGE_H__ */

> 
> when the failure happens, there are messages like this in kmsg:
> 
> mlx5_core 0000:c1:00.0 mlx-p0: entered allmulticast mode
> mlx5_core 0000:c1:00.0 mlx-p0: left allmulticast mode
> mlx5_core 0000:c1:00.0 mlx-p0: failed (err=-22) to set attribute (id=6)
> 
> I am wondering if this is a bug in the current driver or anything above is wrong?
> 
> Some additional information:
> 
> (Fedora stock kernel 6.14.2-300.fc42.x86_64)
> $ uname -a
> Linux epyc-server 6.14.2-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 21:50:55 UTC 2025 x86_64 GNU/Linux
> $ rpm -q iproute
> iproute-6.12.0-3.fc42.x86_64
> 
> Best,
> Qiyu
> 
>