SR-IOV VF NIC with bridge

[CSon4]

Hello all,

I have a problem about SR-IOV.

I use Xen and SR-IOV NIC. Instead of using passthrough NIC, I want to use VF NIC for bridge like

------+ (eth0-PF) +------

| (eth2-VF) |

| | |

| --+------+--------(br0)|

| | |

| (vif1.0) |

| domU |

---------------------------------

In this case, outgoing traffic of domU is passed by vif1.0 --> br0 --> eth2.

The problem is eth0 of domU has different MAC address compared with eth2's. and the traffic of domU is not passed in eth2. My guess is that eth2(VF) drops ethernet frame which has different MAC address.

Is there any one how to solve this problem.

Thanks,

[Patrick_K_Intel1]

Welcome to the blog!

I think I may have answered this one before, try taking a peek at this:

/thread/38613 https://communities.intel.com/thread/38613

If that doesn't help, we will try to reproduce your situation. In order to do that please provide the following information:

1. Are you using Citrix Xenserver or Open Source Xen?

2. Which distrubution (if Open Source Xen)

3. Which version of drivers are you using?

4. Can you provide a Kernel log?

Please look at the 1st thread and see if that helps. If not then if you provide that data, we can look into reproducing you setup.

- Patrick

[CSon4]

Thanks for quick reply.

My environment is a little different from previous thread.

My question can be summarized as

1) Can I modify L2 Sorter in SR-IOV NIC?

2) Can I add additional MAC addresses in MAC Filter in VF NIC?

Thanks.

Choonho Son

[Patrick_K_Intel1]

You can't 'modify the L2 Sorter. It can be 'programmed' to route traffic based on MAC and or VLAN.

You can add more than one MAC address to a VF - which you would need to do if you are using it as a bridge.

However the ability to do this depends on what OS you are using. Frustrating but true that different hypervisors do things differently.

If you have open source Xenserver, the following will work - if you have something different, it likley wont. The following is from my SR-IOV guru:

Assumptions:

1. Opensource Xenserver
2. A standard non-modified Linux distribution
3. Intel 10Gb Network Linux drivers from sourceforge site.
4. Latest bridgeutils and iproute2 software packages.

Steps:

1. Load Intel 10Gb driver with max_vfs=2,2. This will create two VF per port.
2. Load Intel 10Gb VF driver. This will create ethX, ethY, ethZ and ethW. Once eth devicefor each VF.
3. Execute "ip link show" command to ensure each VF has a MAC address assigned.
4. Execute "brctl addbr bridgename". On my setup I have decided to use testbr as a bridge name. So on my setup the command looks like "brctl addbr testbr"
5. Add the VF device to the bridge by executing the command "brctl addif bridgename ethdevice". On my setup eth7 is the VF interface name given by Linux. So on my setup the command looks like "brctl addif testbr eth7"
6. You can also add an ip address to the bridge interface to make sure it is working. To do that"ip addr add ipaddress/mask broadcast ipaddress dev bridgename". On my setup the command looks like "ip addr add 192.168.133.1/24 broadcast 192.168.133.255 dev testbr".
7. Now you should be able to ping from your host to another host on the network within samebroadcast domain.
8. You can create a VM and have the VM Network adapter connect to the newly created bridge. Alltraffic from the VM will flow over the bridge via VF to the physical address.

[CSon4]

Patrick! Thank you for testing.

Your Step is same with me. I succeeded until Step 7. The problem occurred in Step 8.

Bridge has same mac address with VF's MAC, so it can communicate with internet. But VM's MAC is different from VF's MAC. The outgoing traffic from VM to internet via bridge and VF is working(with spoofchk off). But incoming traffic is problem. The destination MAC address of incoming packet is VM's MAC not VF MAC, so L2 Sorter miss-delivers packets to PF NIC instead of VF NIC. Since L2 Sorter does not know VM's MAC address.

For my environment, I use I350-T2, 1G SR-IOV NIC with opensource Xen 4.3 and Ubuntu 12.04 Dom0.

PF driver(eth0): igb-5.0.6

VF driver(eth2): igbvf-2.0.2-k

eth0 MAC : a0:36:9f:15:a8:7c

eth2 MAC : b6:0d:13:18:c6:97

testbr MAC: b6:0d:13:18:c6:97

VM MAC: 00:16:3e:40:d8:29

ip link set eth0 vf 1 spoofchk off

Thanks,

Choonho Son

[CSon4]

Hello Patrick.

You can add more than one MAC address to a VF - which you would need to do if you are using it as a bridge.

==> Could you explain how to add additional MAC address? or related document?

Thanks,

Choonho Son

[Patrick_K_Intel1]

You can add more MAC address to a VF using the iproute2 utility - make sure you have all the latest goodies, kernel, drivers, uproute2 etc.

The VF will show up as an eth device in your OS, use the iproute2 utility to assign a mac address to the the device. Alog the lines for the following (though you will need to consult the docs for exact syntax):

ip link add link eth0 dev peth0 type macvlan address aa:aa:aa:aa:aa:aa

ip link add link eth0 dev peth1 type macvlan address bb:bb:bb:bb:bb:bb

ip link add link eth0 dev peth2 type macvlan address cc:cc:cc:cc:cc:cc

Assuming eth0 was you VF, that will assign 3 logical interfaces to the VF with the given MAC addresses.

Another way is to use the command a bit differently:

ip link add ethX address CA:FE:FE:ED:00:10 ethX.1 type macvlan

ip link add ethX address CA:FE:FE:ED:00:11 ethX.2 type macvlan

ip link add ethX address CA:FE:FE:ED:00:12 ethX.3 type macvlan

This will add 3 MAC addresses to the VF assigned to ethX, and it will create 3 vtap interfaces that can be assigned to a VM.

One thing to remember is that a bridge actually requires the Ethernet device assigned to it to be in promiscous mode - which is NOT supported by a VF. So you can't actually use a traditional bridge. What you can do is use the above to assign multiple MAC addresses to a VF, and then the resulting additional interfaces can be assigned to VMs.

Hope this helps,

Patrick