netdev - Re: SR-IOV + switchdev + vlan + Mellanox: Cannot ping

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ZizS4MlZcIE0KoHq@nanopsycho>
Date: Sat, 27 Apr 2024 12:26:40 +0200
From: Jiri Pirko <jiri@...nulli.us>
To: Shane Miller <gshanemiller6@...il.com>
Cc: netdev@...r.kernel.org
Subject: Re: SR-IOV + switchdev + vlan + Mellanox: Cannot ping

Fri, Apr 26, 2024 at 10:35:28PM CEST, gshanemiller6@...il.com wrote:
>Problem:
>-----------------------------------------------------------------
>root@...hA $ ping 10.xx.xx.194
>PING 10.xx.xx.194 (10.xx.xx.194) 56(84) bytes of data
>>From 10.xx.xx.191 icmp seq=10 Destination Host Unreachable
>Proximate Cause:
>-----------------------------------------------------------------
>This seems to be a side effect of "switchdev" mode. When the identical
>configuration is set up EXCEPT that the SR-IOV virtualized NIC is left
>"legacy", ping (and ncat) works just fine.
>
>As far as I can tell I need a bridge or bridge commands, but I have no
>idea where to start. This environment will not allow me to add modify
>commands when enabling switchdev mode. devlink seems to accept
>"switchdev" alone without modifiers.

You have to configure forwarding between appropriate representors. Use
ovs (probably easiest) or tc.

>
>Note: putting a NIC into switchdev mode makes the virtual functions
>show as "link-state disable" which is confusing. (See below.) Contrary
>to what it seems to suggest, the virtual NICs are up and running
>
>Running "arp -e" on machine A shows machine B's ieth3v0 MAC address as
>incomplete suggesting switchdev+ARP is broken.
>
>Problem Environment:
>-----------------------------------------------------------------
>OS: RHEL 8.6 4.18.0-372.46.1.el8 x64
>NICs: Mellanox ConnectX-6
>
>Machine A Links:
>70 tst@...h3: <...LOWER_UP...> mtu 1500
>   link/ether xx.xx.xx.xx.xx.xx
>   vlan protocol 802.1Q id 133 <REORDER_HDR>
>   Inet 10.xx.xx.191
>
>Machine B Links With ieth3 in SR-IOV mode in switchdev mode:
># Physical Function and its virtual functions:
>                                                 2: ieth3:
><...PROMISC,UP,LOWER_UP> mtu 1500
>    link/ether xx.xx.xx.xx.xx.f6 portname p0 switchid xxxxe988
>    vf 0 link/ether xx.xx.xx.xx.xx.00 vlan 133 spoof off, link-state
>disable, trust off
>    . . .
># Port representers
>893: ieth3r0: <...UP,LOWER_UP> mtu 1500
>link/ether xx.xx.xx.xx.xx.e1 portname pf0vf0 switchid xxxxe988
>. . .
># Virtual Links
>897: ieth3v0: <...UP,LOWER_UP> mtu 1500
>  link/ether xx.xx.xx.xx.xx.00 promiscuity 0
>  inet 10.xx.xx.194/24 scope global ieth3v0
>  . . .
>
>SR-IOV Setup Summary
>-----------------------------------------------------------------
>This is done right since, in legacy mode, ping/ncat works fine:
>
>1. Enable IOMMU, Vtx in BIOS
>2. Boot Linux with iommu=on on command line
>3. Install Mellanox OFED
>4. Enable SR-IOV for max 8 devices in Mellanox firmware
>(reboot)
>5. Create 4 virtual NICs w/ SR-IOV
>6. Configure 4 virtual NICs mac, trust off, spoofchk off, state auto
>7. Unbind virtual NICs
>8. Put ieth3 into switchdev mode
>9. Rebind virtual NICs
>10. Bring all links up
>11. Assign IPV4 addresses to virtual links
>