SR-IOV on 82599EB 10G-Cards: Packets not filtered by MAC Address

[RNeum1]

Hi everybody,

we are trying to set up FPP using Intel's 82599EB 10-Gbit/s Cards.

Furthermore we are using the ixgbe driver version 3.14.5.

I am loading the driver using

modprobe ixgbe max_vfs=4,4,4,4

and bringing up the wanted virtual interfaces using

ifconfig / up

But when sending packets to the interfaces from another machine, setting the destination MAC addresses of the packets alternating to the respective MAC adresses of the different virtual interfaces, all packets will nonetheless be captured by only one of those interfaces.

I checked with tcpdump on this interface, whether the received packets are actually carrying the different destination MAC addresses of my virtual interfaces on the targeted machine. And indeed they do.

Why now doesn't each virtual adapter receive exactly those packets carrying their respective MAC addresses as destination MAC address, but all are being captured by the very same virtual adapter, regardless of their MAC address?

Thank you in advance for your hints and best regards,

Richard

[Patrick_K_Intel1]

Richard,

Thanx for posting to our forum. I encountered the same issue you are explaining when I wrote the FPP papers. I made a mental note to go figure out what the issue was, however I changed jobs before I ever root caused it.

I suspect it has to do with the actual network stack running on your server where the VF's are instantiated. I think the stack gets a bit confused because all the VF's are on the same subnet.

My solution for this was to put each VF on it's own subnet, or to put it on its own VLAN. I bet if you try that, you will get the results you are looking for.

In the meantime I will go pester my co-worker who owns the SR-IOV technology and ask him to go do some digging.

thanx,

Patrick

[Patrick_K_Intel1]

Richard,

We are going to try to replicate your issue. Can you provide the ixgbevf version for us?

thanx,

Patrick

[RNeum1]

Hi Patrick,

thank you for your quick reply.

I am using ixgbevf version 2.2.0-k.

I am furthermore using the kernel version 3.4.33-2.24 under openSUSE 12.2.

Please don't hesitate to contact me if you need further information.

Best regards,

Richard

[Patrick_K_Intel1]

Thanx for the information.

My expert is going to repro your setup and see what he can figure out. Will keep you posted.

- Patrick

[Patrick_K_Intel1]

We were able to reproduce your issue. Both with SR-IOV and with plain ports.

My expert came back with this:

This behavior has to do with subnet mask, Linux network stack and Linux routing. Basically, if you put multiple network interfaces virtual or physical, Linux OS will use the first available interface. Linux makes no guarantees which interface it will use for transmit or receive. It may use one interface for both transmit and receive. It may use one interface for transmit only and another one for receive only. The behavior is unpredictable. It is an OS issue as opposed to a NIC or driver one.

As I indicated earlier, I had encounterd this when I setup the demos for the FPP. My solution was to put the VF's on different subnets, or to put them on different VLANs.

[RNeum1]

Hello Patrick,

many thanks to you and your expert for your support!

I tried to move the interfaces into different subnets as well as to different VLANs (using vconfig) and tagged the outgoing packets on the the sender with the appropriate VLAN tags by sending them throuch virtual IFs as well.

Unfortunately the problem stays the same.

Maybe you can give me some hint, how you did set up those VLANs and subnets.

Thank you

Richard

[Patrick_K_Intel1]

Hi Richard,

To configure ip address on a VF, simply use the ifconfig:

# ifconfig ethx 192.168.100.1 netmask 255.255.255.0

where ethx is the eth device associated with the VF

to set a VLAN for a vf, you use the iproute2 utility on the VF via the PF:

# ip link set eth2 vf 0 vlan 123

This sets VF 0 on the PF assigned to eth2 to 123.

There are more examples all available in my whitepaper:

/community/wired/blog/2012/06/25/latest-flexible-port-partitioning-paper-is-now-available-learn-about-qos-and-sr-iov http://communities.intel.com/community/wired/blog/2012/06/25/latest-flexible-port-partitioning-paper-is-now-available-learn-about-qos-and-sr-iov

Good luck!

- Patrick