| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <52FB9950.4000801@univ-nantes.fr>
Date: Wed, 12 Feb 2014 16:54:56 +0100
From: Yoann Juet <yoann.juet@...v-nantes.fr>
To: Ariel Elior <ariele@...adcom.com>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: bnx2x + SR-IOV, no internal L2 switching
Hi Ariel,
> [Q1]: Are you attaching physical functions to VMs? I.e. are you passing through PFs to the VMs, or only virtual functions?
I'm only attaching VF to the VMs.
> [Q2]: Are you using VFs only on VMs, or directly from the Hypervisor too?
VF are only used on VMs.
> [Q3]: You wrote "for instance the physical ethX has an IP address". That in itself is no problem and no surprise. You gave this as an example to traffic arriving where is shouldn't. Please elaborate.
See below.
> [Q4]: Do you have vlan filters configured anywhere in your topology? Are they configured from Host or from Guest?
No, currently all my tests are conducted without vlan isolation.
> [Q5]: In the case you mentioned where you saw in a VM traffic which was destined to another VM: Did both VMs contain VFs? Were the VFs created from the same Physical Function? If not, what were the BDFs pf the respective PFs? Which mac addresses did you give the VFs?
Each VMs has it's own VF from the same PF. Before starting VMs, I set
the MAC address of each VF with a command similar to:
"ip link set eth0 vf <vf_num> mac de:ad:be:ff:ff:<num>"
>
> [Q6]: Please isolate a specific case where switching is not behaving as expected and describe it in more detail:
> Please describe the topology (which PFs are involved and which VFs. Who is assigned to which VMs)
|PF| ----|VF0 de:ad:be:ff:ff:11| ---- |VM1|
|--|VF1 de:ad:be:ff:ff:12| ---- |VM2|
----
HOST
----
the PF (eth0) and VFs attached to it (eth2 -> eth9)
# ip add
....
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
group default qlen 1000
link/ether a4:1f:72:d0:b3:9f brd ff:ff:ff:ff:ff:ff
inet 172.20.6.229/24 brd 172.20.6.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a61f:72ff:fed0:b39f/64 scope link
valid_lft forever preferred_lft forever
...
30: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:11 brd ff:ff:ff:ff:ff:ff
31: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:12 brd ff:ff:ff:ff:ff:ff
32: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:13 brd ff:ff:ff:ff:ff:ff
33: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:14 brd ff:ff:ff:ff:ff:ff
34: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:15 brd ff:ff:ff:ff:ff:ff
35: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:16 brd ff:ff:ff:ff:ff:ff
36: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:17 brd ff:ff:ff:ff:ff:ff
37: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether de:ad:be:ff:ff:18 brd ff:ff:ff:ff:ff:ff
# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether a4:1f:72:d0:b3:9c brd ff:ff:ff:ff:ff:ff promiscuity 0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
mode DEFAULT group default qlen 1000
link/ether a4:1f:72:d0:b3:9f brd ff:ff:ff:ff:ff:ff promiscuity 0
vf 0 MAC de:ad:be:ff:ff:11, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 1 MAC de:ad:be:ff:ff:12, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 2 MAC de:ad:be:ff:ff:13, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 3 MAC de:ad:be:ff:ff:14, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 4 MAC de:ad:be:ff:ff:15, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 5 MAC de:ad:be:ff:ff:16, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 6 MAC de:ad:be:ff:ff:17, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 7 MAC de:ad:be:ff:ff:18, tx rate 10000 (Mbps), spoof checking
on, link-state auto
4: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default
link/ether 8e:31:e9:eb:89:b9 brd ff:ff:ff:ff:ff:ff promiscuity 0
bond
32: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:13 brd ff:ff:ff:ff:ff:ff promiscuity 0
33: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:14 brd ff:ff:ff:ff:ff:ff promiscuity 0
34: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:15 brd ff:ff:ff:ff:ff:ff promiscuity 0
35: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:16 brd ff:ff:ff:ff:ff:ff promiscuity 0
36: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:17 brd ff:ff:ff:ff:ff:ff promiscuity 0
37: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:18 brd ff:ff:ff:ff:ff:ff promiscuity 0
38: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:11 brd ff:ff:ff:ff:ff:ff promiscuity 0
39: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:12 brd ff:ff:ff:ff:ff:ff promiscuity 0
# lspci
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet (rev 10)
01:09.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.4 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.5 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.6 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
01:09.7 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810
10 Gigabit Ethernet Virtual Function
I'm using libvirt with <hostdev> XML blocks to assign VF0 to VM1 and VF1
to VM2 (also tried <interface type='network'> and network pool, that
does not make a difference):
VM1 is attached to VF0:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x09' function='0x0'/>
</source>
</hostdev>
VM2 is attached to VF1:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x09' function='0x1'/>
</source>
</hostdev>
---------
Once VM1 and VM2 are started, we can see:
----
HOST
----
# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
2: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether a4:1f:72:d0:b3:9c brd ff:ff:ff:ff:ff:ff promiscuity 0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
mode DEFAULT group default qlen 1000
link/ether a4:1f:72:d0:b3:9f brd ff:ff:ff:ff:ff:ff promiscuity 0
vf 0 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 1 MAC 00:00:00:00:00:00, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 2 MAC de:ad:be:ff:ff:13, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 3 MAC de:ad:be:ff:ff:14, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 4 MAC de:ad:be:ff:ff:15, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 5 MAC de:ad:be:ff:ff:16, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 6 MAC de:ad:be:ff:ff:17, tx rate 10000 (Mbps), spoof checking
on, link-state auto
vf 7 MAC de:ad:be:ff:ff:18, tx rate 10000 (Mbps), spoof checking
on, link-state auto
4: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default
link/ether 8e:31:e9:eb:89:b9 brd ff:ff:ff:ff:ff:ff promiscuity 0
bond
32: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:13 brd ff:ff:ff:ff:ff:ff promiscuity 0
33: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:14 brd ff:ff:ff:ff:ff:ff promiscuity 0
34: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:15 brd ff:ff:ff:ff:ff:ff promiscuity 0
35: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:16 brd ff:ff:ff:ff:ff:ff promiscuity 0
36: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:17 brd ff:ff:ff:ff:ff:ff promiscuity 0
37: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether de:ad:be:ff:ff:18 brd ff:ff:ff:ff:ff:ff promiscuity 0
---
VM1
---
# ip add
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
qlen 1000
link/ether de:ad:be:ff:ff:11 brd ff:ff:ff:ff:ff:ff
inet 172.20.6.221/24 brd 172.20.6.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::dcad:beff:feff:ff11/64 scope link
valid_lft forever preferred_lft forever
---
VM2
---
# ip add
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
qlen 1000
link/ether de:ad:be:ff:ff:12 brd ff:ff:ff:ff:ff:ff
inet 172.20.6.222/24 brd 172.20.6.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::dcad:beff:feff:ff12/64 scope link
valid_lft forever preferred_lft forever
> Where are you sending data from (a VF, a PF, some peer on the
network) and where is it arriving (perhaps arriving in multiple places)
> Please detail the behavior you expect and the behavior you observe.
When I send packets from VM1 to the default router 172.20.6.1 (this is
an external device) I observe that all the traffic is visible on VM2
(tshark listening on eth0):
vm2# tshark -i eth0 icmp
...
0.000000 172.20.6.221 -> 172.20.6.1 ICMP 98 Echo (ping) request
id=0x09ee, seq=1/256, ttl=64
0.000297 172.20.6.1 -> 172.20.6.221 ICMP 98 Echo (ping) reply
id=0x09ee, seq=1/256, ttl=64
Same result when permuting VM1 and VM2. VM2 sees all traffic
received/sent to VM1.
Another test, I send packets from the outside to the PF (172.20.6.229)
-> VM1 and VM2 see inbound packets as well as responses from the PF.
> Please supply:
> Mac addresses, ip addresses and masks, configured vlans (if any),
promiscuous setting, etc.
> ip -d -d link show on hypervisor and each of the VMs involved
> ethtool -d from Hypervisor PFs
> dmesg from hypervisor
dmesg is attached. I send to you personally the output of ethtool
command (>1MB).
>
> Please note we recently submitted some fixes to our tx-switching behavior:
> In e8379c79 "bnx2x: fix VLAN configuration for VFs" we fixed an issue where traffic with the wrong vlan could still end up in a VM configured to a different vlan (hence my questions on vlans).
> In c14db202 "bnx2x: Correct default Tx switching behavior" we fixed a connectivity issue with pf to vf connectivity.
> Depending on your answers to the above, perhaps these might be relevant to your case.
I don't think this is relevant for tx-switching since the issue is also
with VF to VF traffic + concerns RX and TX switching.
Thanks for your assistance,
Regards,
--
Université de Nantes - Direction des Systèmes d'Information
Download attachment "dmesg.gz" of type "application/gzip" (18057 bytes)
View attachment "yoann_juet.vcf" of type "text/x-vcard" (366 bytes)
Download attachment "smime.p7s" of type "application/pkcs7-signature" (3256 bytes)
Powered by blists - more mailing lists