In our setup we have a physical server hosting several virtual machine guests.
All guests are assigned two interfaces which are bonded together for redundancy support.
GUESTS communicate over the native VLAN (bond0) and one or more extra vlan interfaces (bond0.x).
The setup is build with
It took some effort to get this setup up and running. Especially the bonding part. I would like to share our experience for the benefit of the community. I'm not a specialist in this domain and I have no experience with other setups (e.g. other hardware or OS).
I skip the process to enable SR-IOV in the Bios. This will most likely be different on your platform anyhow. See the HP document referenced below for details on HP proliant servers.
We use kernel kernel-2.6.32-431.20.3.el6.x86_64
The kernel parameters intel_iommu=on pci=realloc intremap=no_x2apic_optout are required to enable SR-IOV support.Driver
We upgraded to the latest driver officially supported by HP. Without this driver update communication between the VM and the HOST was only possible on the native network and not on a VLAN tagged network.
The kernel module parameters used are:
options ixgbe max_vfs=63,63
Virsh has support for SR-IOV and can assign a virtual function to the VM. On the HOST two networks are defined, one for eth0 and one for eth1.They use mode=hostdev which is for SR-IOV support. Below is the definition for eth0. The one for eth1 is similar.
Assign two network interfaces to the GUEST, one from passthrough_eth0 and one from passthrough_eth1.
Bonding on the HOST
This was the most tricky part to get right.
Starting with the HOST. We rely on the link state of the interface to trigger the failover. eth0 is the preferred interface. This is to make sure that eth0 is active when possible. The updelay of 30 seconds is the time needed by the switch port to come in the spanning tree forwarding state. Therefore we wait 30 seconds before using eth0 as the active interface when it becomes (again) available. The resulting bonding configuration stored in /etc/modprobe.d/bonding.conf is:
alias bond0 bonding
options bonding mode=1 miimon=100 primary=eth0 updelay=30000
Bonding on the GUEST
Take into account:
There are two possible solutions for this issue (or maybe there are more that I don't know about)
I took this to our SR-IOV Guru, and he came back with this for you:
Problem description leads me to believe customer is unable to communicate between VM on a bridged network and VM with SR-IOV. I believe the packets from SR-IOV VM are dropped by the bridged interface. Workaround for this case was added to Linux kernel 3.4.x. Below is the excerpt of the actual scenario.
In SR-IOV enabled adapters, Physical Function (PF) does not work in bridge mode. When a bridge is created on the PF device, an emulation device in the Virtual Machine (VM) connects to this bridge cannot receive any unicast packets.
To avoid this from occurring, for each emulated device (virtio device in the VM) added to the bridge, the MAC address of the emulated device (virtio device in the VM) needs to be added manually to the forwarding database (FDB) filter table using the iproute2 package bridge tool. This can be done by executing the following command:
# bridge fdb add dev
To display if the mac address is added to the FDB table
# bridge fdb show dev < PF device interface >
The FDB entry when the emulated device is no longer in use or the guest to which it is assigned is moved to a different host must be deleted using this
# bridge fdb del dev
Please instruct the customer to update the Linux kernel to version 3.4.x or newer and the commands above. I'd also recommend disabling VF Anti-spoofing if Linux Channel bonding driver is being used. Below is the command to disable it.
ip link set vf spoofchk off
Example: ip link set p5p2 vf 1 spoofchk off Disabling Spoofchk feature on VF # 1 for Ethernet interface P5P2.
Hope this helps
I'm also interested in bonding inside virtual machine with SR-IOV VF as NIC's.
I see same behavior. Is there any solution to propagate MAC address change to the HOST? Because we have requirement to be able change MAC address of the NIC inside VM.
I have spoofchk disabled on the VF. There aren't any log messages about spoofed packets but traffic doesn't flow through interface after MAC change inside VM. MAC change works if MAC changed synchronously in VM and on VF in HOST.