- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've run into a one-way transmission issue with FreeBSD and Virtual Functions on an Intel 10GbE X520. So far we've tested:
- Ubuntu 12.04 LTS
- CentOS 6.5
- pfSense 2.1 / FreeBSD 8.3
- pfSense 2.2 / FreeBSD 10.0 (alpha)
- FreeBSD 8.3-STABLE
All are instances are running on top of KVM installed on Ubuntu 14, hosted on Intel Grizzly Pass OEM hardware with the above-mentioned card installed. The card's two 10GbE ports are connected back-to-back.
In our test case, all instances above have a VF inside VLAN 100 and are bridged via the 82599 controller's internal L2 'switch.' The Ubuntu and CentOS instances can reach each other on their VF/VLAN100 interfaces, but any FreeBSD instance spun up cannot. tcpdump on a CentOS instance and a FreeBSD instance show the FreeBSD instance send an ARP request, the CentOS instance receive it and respond, but it never reaches the FreeBSD instance. Since this is not an issue on Ubuntu or CentOS, and has persisted across two versions of FreeBSD and two Intel VF driver versions, I have to assume it's unique to FreeBSD. The examples below are between FreeBSD 8.3-STABLE and CentOS 6.5:
freebsd1# tcpdump -i ix0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes
21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel
[root@centos1 ~]# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
^C
24 packets captured
24 packets received by filter
0 packets dropped by kernel
freebsd1# uname -a
FreeBSD freebsd1 8.3-RELEASE FreeBSD 8.3-RELEASE # 0: Mon Apr 9 21:23:18 UTC 2012 mailto:root@mason.cse.buffalo.edu root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
freebsd1# dmesg | grep Virtual
CPU: QEMU Virtual CPU version 2.0.0 (2793.29-MHz K8-class CPU)
ix0: mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0
freebsd1# ifconfig ix0
ix0: flags=8843 metric 0 mtu 1500
options=401bb
ether 52:54:00:44:99:6c
inet6 fe80::5054:ff:fe44:996c%ix0 prefixlen 64 scopeid 0x3
inet 192.168.100.253 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=3
media: Ethernet autoselect
status: active
Our issue is identical in description to https://forums.freebsd.org/viewtopic.php?t=10248 this post. However, there is no "Simulated MSI Support" option in our BIOS and I believe that particular option was unique to his board based on Intel's BIOS release notes. http://permalink.gmane.org/gmane.linux.drivers.e1000.devel/13809 Another individual ran into a similar problem with FreeBSD 10.0.
Note that pfSense runs v1.1.4 of the Intel VF driver. If you http://www.intel.com/support/network/adapter/pro100/sb/CS-031492.htm# 4 look here, Intel provides a link to e1000 FreeBSD VF drivers. However, no VF-specific drivers are mentioned for ixgbe. There is _vf.h and _vf.c source code in the FreeBSD...
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I spoke with my SR-IOV expert and he came back with the following:
- Can FreeBSD VM communicate with other VMs without VLAN configured?
- Please rerun the test with X520 freebsd driver version 2.5.15. Download URL http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/14688/eng/ixgbe-2.5.15.tar.gz&Lang=eng&Dwnldid=14688 http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/14688/eng/ixgbe-2.5.15.tar.gz&Lang=eng&Dwnldid=14688
- Please make sure to assign a MAC address using IPROUTE2 utility to the VF before assigning the VF to FreeBSD VM.
thanx,
Patrick
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Patrick,
Sorry for the delay in response, I've been out of town for the past week. I had someone try the v2.5.15 drivers in my absence. I spoke to one of them briefly and he mentioned that the drivers were compiled and loaded, but there was no difference in behavior. However, it's unclear whether or not we need VF-specific drivers.
Special note on # 3: we allow kvm to auto-generate a MAC for the VF interfaces. This works well with other guests. Is there a reason why auto-assignment could cause issues for FreeBSD? Just curious.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm having the same issue myself. I'm running a FreeBSD 10.0 (updated to latest) on a XenServer 6.2. The VF (ix0) does show up. I manually assign a MAC address to it, and when I ping a VF on a CentOS 6.5 (running on the same hypervisor), the ARP table does get updated on the CentOS 6.5 side with the MAC address of the FreeBSD VF. However, the ARP reply never makes it back. Also, there is nothing in the output of tcpdump on FreeBSD for ix0.
It seems that https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=14688&lang=eng Intel® Download Center is the latest version of VF driver for FreeBSD. The version on the file says 2.5.25, however, once the driver is compiled, the version shows up as 1.1.4. I thought that there was a later version? I was wondering if somebody has made progress in this regard?

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page