Intel X520 (82599) Virtual Functions on FreeBSD 8.3/10

[SPede]

I've run into a one-way transmission issue with FreeBSD and Virtual Functions on an Intel 10GbE X520. So far we've tested:

• Ubuntu 12.04 LTS
• CentOS 6.5
• pfSense 2.1 / FreeBSD 8.3
• pfSense 2.2 / FreeBSD 10.0 (alpha)
• FreeBSD 8.3-STABLE

All are instances are running on top of KVM installed on Ubuntu 14, hosted on Intel Grizzly Pass OEM hardware with the above-mentioned card installed. The card's two 10GbE ports are connected back-to-back.

In our test case, all instances above have a VF inside VLAN 100 and are bridged via the 82599 controller's internal L2 'switch.' The Ubuntu and CentOS instances can reach each other on their VF/VLAN100 interfaces, but any FreeBSD instance spun up cannot. tcpdump on a CentOS instance and a FreeBSD instance show the FreeBSD instance send an ARP request, the CentOS instance receive it and respond, but it never reaches the FreeBSD instance. Since this is not an issue on Ubuntu or CentOS, and has persisted across two versions of FreeBSD and two Intel VF driver versions, I have to assume it's unique to FreeBSD. The examples below are between FreeBSD 8.3-STABLE and CentOS 6.5:

freebsd1# tcpdump -i ix0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes

21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

^C

13 packets captured

13 packets received by filter

0 packets dropped by kernel

[root@centos1 ~]# tcpdump -i eth1

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes

17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28

17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28

^C

24 packets captured

24 packets received by filter

0 packets dropped by kernel

freebsd1# uname -a

FreeBSD freebsd1 8.3-RELEASE FreeBSD 8.3-RELEASE # 0: Mon Apr 9 21:23:18 UTC 2012 mailto:root@mason.cse.buffalo.edu root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

freebsd1# dmesg | grep Virtual

CPU: QEMU Virtual CPU version 2.0.0 (2793.29-MHz K8-class CPU)

ix0: mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0

freebsd1# ifconfig ix0

ix0: flags=8843 metric 0 mtu 1500

options=401bb

ether 52:54:00:44:99:6c

inet6 fe80::5054:ff:fe44:996c%ix0 prefixlen 64 scopeid 0x3

inet 192.168.100.253 netmask 0xffffff00 broadcast 192.168.100.255

nd6 options=3

media: Ethernet autoselect

status: active

Our issue is identical in description to https://forums.freebsd.org/viewtopic.php?t=10248 this post. However, there is no "Simulated MSI Support" option in our BIOS and I believe that particular option was unique to his board based on Intel's BIOS release notes. http://permalink.gmane.org/gmane.linux.drivers.e1000.devel/13809 Another individual ran into a similar problem with FreeBSD 10.0.

Note that pfSense runs v1.1.4 of the Intel VF driver. If you http://www.intel.com/support/network/adapter/pro100/sb/CS-031492.htm# 4 look here, Intel provides a link to e1000 FreeBSD VF drivers. However, no VF-specific drivers are mentioned for ixgbe. There is _vf.h and _vf.c source code in the FreeBSD...

[Patrick_K_Intel1]

I spoke with my SR-IOV expert and he came back with the following:

1. Can FreeBSD VM communicate with other VMs without VLAN configured?
2. Please rerun the test with X520 freebsd driver version 2.5.15. Download URL http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/14688/eng/ixgbe-2.5.15.tar.gz&Lang=eng&Dwnldid=14688 http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/14688/eng/ixgbe-2.5.15.tar.gz&Lang=eng&Dwnldid=14688
3. Please make sure to assign a MAC address using IPROUTE2 utility to the VF before assigning the VF to FreeBSD VM.

thanx,

Patrick

[SPede]

Hi Patrick,

Sorry for the delay in response, I've been out of town for the past week. I had someone try the v2.5.15 drivers in my absence. I spoke to one of them briefly and he mentioned that the drivers were compiled and loaded, but there was no difference in behavior. However, it's unclear whether or not we need VF-specific drivers.

Special note on # 3: we allow kvm to auto-generate a MAC for the VF interfaces. This works well with other guests. Is there a reason why auto-assignment could cause issues for FreeBSD? Just curious.

[SB13]

I'm having the same issue myself. I'm running a FreeBSD 10.0 (updated to latest) on a XenServer 6.2. The VF (ix0) does show up. I manually assign a MAC address to it, and when I ping a VF on a CentOS 6.5 (running on the same hypervisor), the ARP table does get updated on the CentOS 6.5 side with the MAC address of the FreeBSD VF. However, the ARP reply never makes it back. Also, there is nothing in the output of tcpdump on FreeBSD for ix0.

It seems that https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=14688&lang=eng Intel® Download Center is the latest version of VF driver for FreeBSD. The version on the file says 2.5.25, however, once the driver is compiled, the version shows up as 1.1.4. I thought that there was a later version? I was wondering if somebody has made progress in this regard?