netdev - [question] when bonding with CX5 network card that support ROCE

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <756aaf3c-5a15-8d18-89d4-ea7380cf845d@huawei.com>
Date: Mon, 6 May 2024 12:46:55 +0800
From: shaozhengchao <shaozhengchao@...wei.com>
To: <saeedm@...dia.com>, <tariqt@...dia.com>, <borisp@...dia.com>,
	<shayd@...dia.com>, <msanalla@...dia.com>, Rahul Rameshbabu
	<rrameshbabu@...dia.com>, <weizhang@...dia.com>, <kliteyn@...dia.com>,
	<erezsh@...dia.com>, <igozlan@...dia.com>
CC: netdev <netdev@...r.kernel.org>, <linux-rdma@...r.kernel.org>
Subject: [question] when bonding with CX5 network card that support ROCE


When using the 5.10 kernel, I can find two IB devices using the 
ibv_devinfo command.
----------------------------------
[root@...alhost ~]# lspci
91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family 
[ConnectX-5]
91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
----------------------------------
[root@...alhost ~]# ibv_devinfo
hca_id: mlx5_0
         transport:                      InfiniBand (0)
         fw_ver:                         16.31.1014
         node_guid:                      f41d:6b03:006f:4743
         sys_image_guid:                 f41d:6b03:006f:4743
         vendor_id:                      0x02c9
         vendor_part_id:                 4119
         hw_ver:                         0x0
         board_id:                       HUA0000000004
         phys_port_cnt:                  1
                 port:   1
                         state:                  PORT_ACTIVE (4)
                         max_mtu:                4096 (5)
                         active_mtu:             1024 (3)
                         sm_lid:                 0
                         port_lid:               0
                         port_lmc:               0x00
                         link_layer:             Ethernet

hca_id: mlx5_1
         transport:                      InfiniBand (0)
         fw_ver:                         16.31.1014
         node_guid:                      f41d:6b03:006f:4744
         sys_image_guid:                 f41d:6b03:006f:4743
         vendor_id:                      0x02c9
         vendor_part_id:                 4119
         hw_ver:                         0x0
         board_id:                       HUA0000000004
         phys_port_cnt:                  1
                 port:   1
                         state:                  PORT_ACTIVE (4)
                         max_mtu:                4096 (5)
                         active_mtu:             1024 (3)
                         sm_lid:                 0
                         port_lid:               0
                         port_lmc:               0x00
                         link_layer:             Ethernet
----------------------------------
But after the two network ports are bonded, only one IB device is
available, and only PF0 can be used.
[root@...alhost shaozhengchao]# ibv_devinfo
hca_id: mlx5_bond_0
         transport:                      InfiniBand (0)
         fw_ver:                         16.31.1014
         node_guid:                      f41d:6b03:006f:4743
         sys_image_guid:                 f41d:6b03:006f:4743
         vendor_id:                      0x02c9
         vendor_part_id:                 4119
         hw_ver:                         0x0
         board_id:                       HUA0000000004
         phys_port_cnt:                  1
                 port:   1
                         state:                  PORT_ACTIVE (4)
                         max_mtu:                4096 (5)
                         active_mtu:             1024 (3)
                         sm_lid:                 0
                         port_lid:               0
                         port_lmc:               0x00
                         link_layer:             Ethernet

The current Linux mainline driver is the same.

I found the comment ("If bonded, we do not add an IB device for PF1.")
in the mlx5_lag_intf_add function of the 5.10 branch driver code.
This indicates that wthe the same NIC is used, only PF0 support bonding?
Are there any other constraints, when enable bonding with CX5?

Thank you
Zhengchao Shao