netdev - RE: SR-IOV + switchdev + vlan + Mellanox: Cannot ping

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAFtQo5D8D861XjUuznfdBAJxYP1wCaf_L482OuyN4BnNazM1eg@mail.gmail.com>
Date: Fri, 26 Apr 2024 16:35:28 -0400
From: Shane Miller <gshanemiller6@...il.com>
To: netdev@...r.kernel.org
Subject: RE: SR-IOV + switchdev + vlan + Mellanox: Cannot ping

Problem:
-----------------------------------------------------------------
root@...hA $ ping 10.xx.xx.194
PING 10.xx.xx.194 (10.xx.xx.194) 56(84) bytes of data
>From 10.xx.xx.191 icmp seq=10 Destination Host Unreachable
Proximate Cause:
-----------------------------------------------------------------
This seems to be a side effect of "switchdev" mode. When the identical
configuration is set up EXCEPT that the SR-IOV virtualized NIC is left
"legacy", ping (and ncat) works just fine.

As far as I can tell I need a bridge or bridge commands, but I have no
idea where to start. This environment will not allow me to add modify
commands when enabling switchdev mode. devlink seems to accept
"switchdev" alone without modifiers.

Note: putting a NIC into switchdev mode makes the virtual functions
show as "link-state disable" which is confusing. (See below.) Contrary
to what it seems to suggest, the virtual NICs are up and running

Running "arp -e" on machine A shows machine B's ieth3v0 MAC address as
incomplete suggesting switchdev+ARP is broken.

Problem Environment:
-----------------------------------------------------------------
OS: RHEL 8.6 4.18.0-372.46.1.el8 x64
NICs: Mellanox ConnectX-6

Machine A Links:
70 tst@...h3: <...LOWER_UP...> mtu 1500
   link/ether xx.xx.xx.xx.xx.xx
   vlan protocol 802.1Q id 133 <REORDER_HDR>
   Inet 10.xx.xx.191

Machine B Links With ieth3 in SR-IOV mode in switchdev mode:
# Physical Function and its virtual functions:
                                                 2: ieth3:
<...PROMISC,UP,LOWER_UP> mtu 1500
    link/ether xx.xx.xx.xx.xx.f6 portname p0 switchid xxxxe988
    vf 0 link/ether xx.xx.xx.xx.xx.00 vlan 133 spoof off, link-state
disable, trust off
    . . .
# Port representers
893: ieth3r0: <...UP,LOWER_UP> mtu 1500
link/ether xx.xx.xx.xx.xx.e1 portname pf0vf0 switchid xxxxe988
. . .
# Virtual Links
897: ieth3v0: <...UP,LOWER_UP> mtu 1500
  link/ether xx.xx.xx.xx.xx.00 promiscuity 0
  inet 10.xx.xx.194/24 scope global ieth3v0
  . . .

SR-IOV Setup Summary
-----------------------------------------------------------------
This is done right since, in legacy mode, ping/ncat works fine:

1. Enable IOMMU, Vtx in BIOS
2. Boot Linux with iommu=on on command line
3. Install Mellanox OFED
4. Enable SR-IOV for max 8 devices in Mellanox firmware
(reboot)
5. Create 4 virtual NICs w/ SR-IOV
6. Configure 4 virtual NICs mac, trust off, spoofchk off, state auto
7. Unbind virtual NICs
8. Put ieth3 into switchdev mode
9. Rebind virtual NICs
10. Bring all links up
11. Assign IPV4 addresses to virtual links