| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <819d5e7f-2e93-4c14-8ac8-dfbba0b9457a@LBSD.net>
Date: Mon, 14 Apr 2025 00:06:12 +0000
From: Nigel Kukard <nkukard@...D.net>
To: netdev@...r.kernel.org
Subject: Problem trying to test a ConnectX-4 LC with switchdev
Hi there,
I'm hoping someone can point out what I'm doing wrong when trying to
configure switchdev mode with a Mellanox ConnectX-4 LC adapter.
I also see that the dmesg output states the normal SRIOV mode is listed
as "LEGACY", does this mean it could be removed in future?
For testing switchdev I'm getting the below error:
Error: mlx5_core: Error checking for existing bridge with same ifindex.
I'm testing out on Archlinux on a clean install.
# uname -a
Linux localhost.localdomain 6.14.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu,
10 Apr 2025 18:43:59 +0000 x86_64 GNU/Linux
# ip link
...
2: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether ec:0d:9a:3a:ce:ca brd ff:ff:ff:ff:ff:ff
altname enp1s0f0np0
altname enxec0d9a3aceca
3: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether ec:0d:9a:3a:ce:cb brd ff:ff:ff:ff:ff:ff
altname enp1s0f1np1
altname enxec0d9a3acecb
...
# lspci -vnn -s 01:00.0; lspci -vnn -s 01:00.1
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx] [15b3:1015]
DeviceName: NIC1
Subsystem: Mellanox Technologies ConnectX-4 Lx 25 GbE Dual Port
SFP28 rNDC [15b3:0025]
Flags: bus master, fast devsel, latency 0, IRQ 126, NUMA node
0, IOMMU group 40
Memory at a6000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b7e00000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1c0] Secondary PCI Express
Capabilities: [230] Access Control Services
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx] [15b3:1015]
DeviceName: NIC2
Subsystem: Mellanox Technologies ConnectX-4 Lx 25 GbE Dual Port
SFP28 rNDC [15b3:0025]
Flags: bus master, fast devsel, latency 0, IRQ 129, NUMA node
0, IOMMU group 41
Memory at a4000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b7f00000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [230] Access Control Services
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
# lspci
...
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx] [15b3:1015]
01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx] [15b3:1015]
01:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx Virtual Function] [15b3:1016]
01:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx Virtual Function] [15b3:1016]
01:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx Virtual Function] [15b3:1016]
01:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx Virtual Function] [15b3:1016]
...
I then unbind from mlx5_core...
# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind
And set the device into switchdev mode...
# devlink dev eswitch set pci/0000:01:00.0 mode switchdev
# devlink dev eswitch show pci/0000:01:00.0
pci/0000:01:00.0: mode switchdev inline-mode link encap-mode basic
# devlink dev show
pci/0000:01:00.0:
nested_devlink:
auxiliary/mlx5_core.eth.0
auxiliary/mlx5_core.eth.0
pci/0000:01:00.1:
nested_devlink:
auxiliary/mlx5_core.eth.1
auxiliary/mlx5_core.eth.1
pci/0000:04:00.0:
nested_devlink:
auxiliary/mlx5_core.eth.2
auxiliary/mlx5_core.eth.2
pci/0000:04:00.1:
nested_devlink:
auxiliary/mlx5_core.eth.3
auxiliary/mlx5_core.eth.3
# dmesg
...
[ 1184.307616] mlx5_core 0000:01:00.0: E-Switch: Disable: mode(LEGACY),
nvfs(4), necvfs(0), active vports(5)
[ 1187.238127] mlx5_core 0000:01:00.0: E-Switch: Tc chains and
priorities offload aren't supported
[ 1187.656245] mlx5_core 0000:01:00.0: sampler_termtbl_create:72:(pid
2002): termination table is not supported
[ 1187.764322] mlx5_core 0000:01:00.0 eno1np0: Dropping C-tag vlan
stripping offload due to S-tag vlan
[ 1187.764330] mlx5_core 0000:01:00.0 eno1np0: Disabling HW_VLAN CTAG
FILTERING, not supported in switchdev mode
[ 1187.905188] mlx5_core 0000:01:00.0 eno1r0: renamed from eth0
[ 1187.997947] mlx5_core 0000:01:00.0 eno1r1: renamed from eth0
[ 1188.094991] mlx5_core 0000:01:00.0 eno1r2: renamed from eth0
[ 1188.186064] mlx5_core 0000:01:00.0: E-Switch: Enable: mode(OFFLOADS),
nvfs(4), necvfs(0), active vports(4)
[ 1188.188408] mlx5_core 0000:01:00.0 eno1r3: renamed from eth0
All looks good so far...
# ip link
...
2: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether ec:0d:9a:3a:ce:ca brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 42:10:90:00:01:00 brd ff:ff:ff:ff:ff:ff, spoof
checking off, link-state disable, trust off, query_rss off
vf 1 link/ether 42:10:90:00:01:01 brd ff:ff:ff:ff:ff:ff, spoof
checking off, link-state disable, trust off, query_rss off
vf 2 link/ether 42:10:90:00:01:02 brd ff:ff:ff:ff:ff:ff, spoof
checking off, link-state disable, trust off, query_rss off
vf 3 link/ether 42:10:90:00:01:03 brd ff:ff:ff:ff:ff:ff, spoof
checking off, link-state disable, trust off, query_rss off
altname enp1s0f0np0
altname enxec0d9a3aceca
...
11: eno1r0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether 2e:4c:3a:17:70:2e brd ff:ff:ff:ff:ff:ff
altname enp1s0f0r0
12: eno1r1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether 46:16:78:95:a5:1f brd ff:ff:ff:ff:ff:ff
altname enp1s0f0r1
13: eno1r2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether 5e:7d:1d:8e:09:d2 brd ff:ff:ff:ff:ff:ff
altname enp1s0f0r2
14: eno1r3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether 66:28:62:9c:2c:59 brd ff:ff:ff:ff:ff:ff
altname enp1s0f0r3
The problem comes in when I try add the device to a bridge... from
networking/device_drivers/ethernet/mellanox/mlx5/switchdev.html
# ip link add testbr0 type bridge
# ip link set eno1np0 master testbr0
Error: mlx5_core: Error checking for existing bridge with same ifindex.
I get the same error trying to add the representor devices...
# ip link set eno1r0 master testbr0
Error: mlx5_core: Error checking for existing bridge with same ifindex.
# dmesg
...
[ 1515.581631] testbr0: port 1(eno1np0) entered blocking state
[ 1515.581637] testbr0: port 1(eno1np0) entered disabled state
[ 1515.581661] mlx5_core 0000:01:00.0 eno1np0: entered allmulticast mode
[ 1515.581724] mlx5_core 0000:01:00.0 eno1np0: left allmulticast mode
[ 1515.581730] mlx5_core 0000:01:00.0 eno1np0: failed (err=-22) to set
attribute (id=6)
[ 1703.825900] testbr0: port 1(eno1r0) entered blocking state
[ 1703.825906] testbr0: port 1(eno1r0) entered disabled state
[ 1703.825937] mlx5_core 0000:01:00.0 eno1r0: entered allmulticast mode
[ 1703.826043] mlx5_core 0000:01:00.0 eno1r0: left allmulticast mode
[ 1703.826051] mlx5_core 0000:01:00.0 eno1r0: failed (err=-22) to set
attribute (id=6)
...
Additional details...
# devlink -V
devlink utility, iproute2-6.14.0
# ip -V
ip utility, iproute2-6.14.0, libbpf 1.5.0
# bridge -V
bridge utility, 6.14.0
# mstflint -d 01:00.0 q
Image type: FS3
FW Version: 14.32.2004
FW Release Date: 13.1.2022
Product Version: 14.32.2004
Rom Info: type=UEFI version=14.25.18 cpu=AMD64
type=PXE version=3.6.502 cpu=IA32
Description: UID GuidsNumber
Base GUID: ec0d9a03003aceca 8
Base MAC: ec0d9a3aceca 8
Image VSD: N/A
Device VSD: N/A
PSID: DEL2810000034
Security Attributes: N/A
Any help or guidance would be greatly appreciated! (please kindly CC me
as I'm not on the list)
Kind Regards
Nigel
Download attachment "OpenPGP_signature.asc" of type "application/pgp-signature" (237 bytes)
Powered by blists - more mailing lists