Bridging Virtual Functions inside a VM

[idata]

Hello all,

I'm looking for some help here. I have a HOST machine with two Intel 82599 ethernet controllers in it. I've created 1 virtual function on each of them and assigned them to a virtual machine. When the virtual machine boots up, it see's the two interfaces as eth0 and eth1. Now I want to bridge eth1 and eth2 so that I can run an Inline Firewall application. After I've done this I've tried to pass traffic and it does not work. I noticed errors in my dmesg output that suggested that the mac spoofing feature was dropping packets. So, I went into the ixgbe_main.c and removed the MAC FILTER and VLAN filtering sections of code and tried again. No more errors in dmesg but still no luck.

Is this possible?

[LAN]--->[Physical Function 1]--->[Virtual Function Guest Eth0]--->[Virtual Function Guest Eth1]---->[Physical Nic Eth2]-->[WAN]

** BRIDGED TOGETHER**

Is there something in the driver I can alter to make this work?

[Patrick_K_Intel1]

If I understand what you are trying to do, yes I believe this should work. Have you done anything with the PF's?

I recently published a paper on using the bonding driver in this situation: /community/wired/blog/2012/06/25/latest-flexible-port-partitioning-paper-is-now-available-learn-about-qos-and-sr-iov http://communities.intel.com/community/wired/blog/2012/06/25/latest-flexible-port-partitioning-paper-is-now-available-learn-about-qos-and-sr-iov

It may have some insight for you.

Can you post the commands you are doing to configure the bridge? If I understand it correctly, you may need to configure the 2nd VF so that the MAC from the 1st is also valid on it.

[idata]

The commands I'm using to create the bridge are as follows:

brctl addbr switch

brctl addif switch eth0

brctl addif switch eth1

ifconfig switch up

And thats it. I can sniff on the hosts ETH4 where I have VF0 and I see an ARP come in. I sniff on the guests eth0 which is mapped to Eth4.VF0 and I see the packet. If I sniff on the guests eth1 I see the arp packet. If I sniff on the hosts Eth2.VF0 which is mapped to the guests eth1 I DO NOT SEE the ARP packet. What I was seeing was Mac Spoofing errors in the logs. So I commented out the MAC spoofing code and I no longer see those errors but the packet still doesnt get through.

You mentioned something about a bonding driver? where would I find that? Is it a replacement for IXGBE driver that I'm using?

[Patrick_K_Intel1]

The bonding driver is a component of the Linux network stack that allows teaming, failover and such on top of ethernet devices. I don't think that is what you are looking to do now that I've donw some research.

I spoke with my experts and they said you need to assign a tap interface to the bridge. These are the steps they gave me (though 've not tried myself), and only shows one interface, however I think it is a starting point for you:

· Linux bridge creation - "brctl addbr bridgename"

· Add interface to Bridge – "brctl addif bridgename interfacename"

· Add tap interface to bridge - "brctl addif bridgename tap0"

· Bring down the Ethernet interface – "ifconfig interfacename down"

· Bring up Ethernet interface – "ifconfig interfacename 0.0.0.0 up"

· Bring up tap interface – "ifconfig tap0 0.0.0.0 up"

· Assign ip address to bridge – "ifconfig bridgename up"

Now create tap interface to unique mac addresses for use.

· Initialize tap interface – "ifconfig tap1 hw ether "

· Assign IP address - "ifconfig tap1 up"

[idata]

All done on the Guest (VM) I assume? I'm a bit puzzled on what the purpose of the TAP is?

I'm trying to get traffic to flow through a bridge on a VM. So what your describing above would look like this?

[PF-ETH2]-->[VF0]--[VM-ETH0]-->[TAP0] --------> [TAP1]--->[VM-ETH1]--[VF0]--[PF-ETH4]

^^^^^^^^^^ALL THIS BRIDGED?? ^^^^^^^^^^^^^^^^^

Lastly, you state I should give a unique mac address tap1 but your example talks about tap0? Do I create another tap?

[Patrick_K_Intel1]

Hmm, my previous post didn't seem to make it. Darn Gremlins - I'll try again.

I went back to my experts and asked them if the following would work for you, they said it should:

brctl addbr switch

brctl addif switch eth0

brctl addif switch eth1

brctl addif switch tap0

ifconfig switch down

ifconfig tap0 0.0.0.0 up

ifconfig switch up

ifconfig tap0 hw ether

ifconfig tap0 0.0.0.0 up

I've not used the MacVtap so I am unable to shed much light on it other than to point you to a wiki: http://virt.kernelnewbies.org/MacVTap http://virt.kernelnewbies.org/MacVTap.

[idata]

Thanks for the help. I feel like I'm making progress. So after doing this I get an error saying VF 0 requested MACVLAN filter but is administratively denied. Is there an option to turn MACVLAN filtering on or off or something? Or am I reading this error message wrong.

[Patrick_K_Intel1]

John - don't thank me yet. Seems as if I've led you down the wrong rabbit hole :-)

I went back once again to my experts and in clear, simple, slow words explained the situation and they realized they were trying to solve the wrong problem before. So long story short, ignore everything I write earlier today (except the link to my whitepaper).

Your steps of:

brctl addbr switch

brctl addif switch eth0

brctl addif switch eth1

ifconfig switch up

Were accurate, but as you figured out, the anti-spoofing features were then causing you issues. You cannot simply go and comment out some code as you mentioned.

You need to disble the anit-spoofing, to do this, you need the latest and greated iproute2 utility, which is located :http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2

After you get that up and running, then for each PF, you must disable the antispoofing for the individual vflike so:

ip link set ethX vf N spoofchk off

Again - that is for the PF, so in your VMM do that.

Give that a try and my apologies for all the other fun stuff I had you do :-)

- Patrick

[idata]

So I installed the latest iproute2 and disabled mac spoofing protection. I still can not get traffic to pass through it. I am no longer getting the error messages in the logs however. What is happening after disabling mac spoofing is that ARPs (broadcast based traffic) and any other broadcast based traffic and multicast traffic gets through our bridge but unicast traffic does not. So, said another way. If mac address ABC talks to FFFFF... that goes through but if ABC talks to EFG it never arrives the interface of the guest (PF never gets it to VF).

Normally on a switch the switch needs to build a MAC table in order to forward traffic. Its acting as if the L2 switch in the 82599 is not learning mac addresses other than broadcast and multicast ones.

[idata]

Any idea what VF 0 requested MACVLAN filter but is administratively denied means and how to turn that capability off?

[Patrick_K_Intel1]

I believe that if you do any kind of configuration in the VMM (such as assigning a MAC or VLAN), for security reasons, the VM is not allowed to make any configuration changes.

This is by design for security - one would not want a VM to change settings the VMM configured.

- Patrick