netdev - Re: [question] when bonding with CX5 network card that support ROCE

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <7b0e61e1-8d50-4431-bd0a-6398c618a609@linux.dev>
Date: Mon, 6 May 2024 10:26:01 +0200
From: Zhu Yanjun <zyjzyj2000@...il.com>
To: shaozhengchao <shaozhengchao@...wei.com>, saeedm@...dia.com,
 tariqt@...dia.com, borisp@...dia.com, shayd@...dia.com, msanalla@...dia.com,
 Rahul Rameshbabu <rrameshbabu@...dia.com>, weizhang@...dia.com,
 kliteyn@...dia.com, erezsh@...dia.com, igozlan@...dia.com
Cc: netdev <netdev@...r.kernel.org>, linux-rdma@...r.kernel.org
Subject: Re: [question] when bonding with CX5 network card that support ROCE

On 06.05.24 06:46, shaozhengchao wrote:
> 
> When using the 5.10 kernel, I can find two IB devices using the 
> ibv_devinfo command.
> ----------------------------------
> [root@...alhost ~]# lspci
> 91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family 
> [ConnectX-5]
> 91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
> ----------------------------------
> [root@...alhost ~]# ibv_devinfo
> hca_id: mlx5_0
>          transport:                      InfiniBand (0)
>          fw_ver:                         16.31.1014
>          node_guid:                      f41d:6b03:006f:4743
>          sys_image_guid:                 f41d:6b03:006f:4743
>          vendor_id:                      0x02c9
>          vendor_part_id:                 4119
>          hw_ver:                         0x0
>          board_id:                       HUA0000000004
>          phys_port_cnt:                  1
>                  port:   1
>                          state:                  PORT_ACTIVE (4)
>                          max_mtu:                4096 (5)
>                          active_mtu:             1024 (3)
>                          sm_lid:                 0
>                          port_lid:               0
>                          port_lmc:               0x00
>                          link_layer:             Ethernet
> 
> hca_id: mlx5_1
>          transport:                      InfiniBand (0)
>          fw_ver:                         16.31.1014
>          node_guid:                      f41d:6b03:006f:4744
>          sys_image_guid:                 f41d:6b03:006f:4743
>          vendor_id:                      0x02c9
>          vendor_part_id:                 4119
>          hw_ver:                         0x0
>          board_id:                       HUA0000000004
>          phys_port_cnt:                  1
>                  port:   1
>                          state:                  PORT_ACTIVE (4)
>                          max_mtu:                4096 (5)
>                          active_mtu:             1024 (3)
>                          sm_lid:                 0
>                          port_lid:               0
>                          port_lmc:               0x00
>                          link_layer:             Ethernet
> ----------------------------------
> But after the two network ports are bonded, only one IB device is
> available, and only PF0 can be used.
> [root@...alhost shaozhengchao]# ibv_devinfo
> hca_id: mlx5_bond_0
>          transport:                      InfiniBand (0)
>          fw_ver:                         16.31.1014
>          node_guid:                      f41d:6b03:006f:4743
>          sys_image_guid:                 f41d:6b03:006f:4743
>          vendor_id:                      0x02c9
>          vendor_part_id:                 4119
>          hw_ver:                         0x0
>          board_id:                       HUA0000000004
>          phys_port_cnt:                  1
>                  port:   1
>                          state:                  PORT_ACTIVE (4)
>                          max_mtu:                4096 (5)
>                          active_mtu:             1024 (3)
>                          sm_lid:                 0
>                          port_lid:               0
>                          port_lmc:               0x00
>                          link_layer:             Ethernet
> 
> The current Linux mainline driver is the same.
> 
> I found the comment ("If bonded, we do not add an IB device for PF1.")
> in the mlx5_lag_intf_add function of the 5.10 branch driver code.

Not sure if rdma lag is enabled for this or not. /proc/net/bonding will 
provide more more details normally.

Zhu Yanjun

> This indicates that wthe the same NIC is used, only PF0 support bonding?
> Are there any other constraints, when enable bonding with CX5?
> 
> Thank you
> Zhengchao Shao