SR-IOV on 82576 on, VM on VF can not talk to VM on Bridged network

[SFors1]

I have set up SR-IOV on a Dell C6145 that has an Intell 82576.

Configuration is

VM1 is assigned a direct VF, on network 10.10.6.0/22

VM2 on same host is not assigned a VF, and has a bridged network br0 to 10.10.6.0/22

VM3 on same host is also on bridged network br0

Separate Physical machine Ph1 is also on 10.10.6.0/22

Ph1 can ping VM1, VM2, VM3

VM1 can ping Ph1 , but not VM2 or VM3

VM2 can ping VM3 and Ph1

VM3 can ping VM2 and Ph1

using tcpdump on VM1, I can see arp broadcast asking for mac address, and arp tables shows it knows the mac address, but no packets are being delivered/routed.

Is there something I am overlooking?

[Patrick_K_Intel1]

Thanx for bringing your question to our blog site.

What you are experiencing is a challenge with SR-IOV. Basically it comes down to the fact that the hardware in the Intel NIC filters traffic going to a VF based upon MAC address and/or VLAN tag. So in your case, VM1 goes to transmit data, and the MAC address for VM2 and VM3 are unknown to the Intel NIC (because they are handled by the bridge), and as such it transmits it out on the wire.

This is why VM1 can communicate with Ph1. Also is why VM2 and VM3 can send data to VM1 – because when the packets reach the Intel NIC, it recognizes the destination MAC address and sends it to the VF.

There is something you can do about this, though it does require some work (until somebody decides to add more support in the kernel to do it in a more automated fashion).

Below are the steps to make VM1 (VF assigned) communicate with VM2 and VM3 (VM2 & VM3 are connected to BR0).

1. Make sure latest Linux distribution is being used. We tested this setup on Fedora 19 64-bit using Intel X520 adapter.
2. Make sure you are using NIC controller drivers that are included in your Linuxdistribution.
3. Assuming eth2 is the port that has VFs. VF from eth2 is assigned to VM1.
4. "brctl addif br0 eth2" ( this will assign eth2 to br0. )
5. "bridge fdb add 'insert mac address for VM2 here' device eth2 self"
6. "bridge fdb add 'insert mac address for VM3 here' device eth2 self"
7. Now you should be able to ping to and from VM1 to VM2 & VM3.

Hope this helps,

Patrick

[AEYLA]

Hi Patrick,

The work-around/solution you propose works well for us in an untagged (no VLAN) network.

However, in a tagged/VLAN network we are are experiencing connectivity issues between VM1 and VM2/3.

VM1 (VF assigned) and VM2/3 (bridge/OVS connected) are both tagged with same VLAN.

Is there a way to populate the MAC address of VM2 and VM3 within a specific VLAN scope?

Environment:

OS: RHEL 7.0

NIC: HP Ethernet 10Gb 2-port 560SFP, Controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ , Driver: ixgbe 3.15.1-k

Thank you, Assaf