Problems with Intel XL710, SR-IOV and Openstack

TBrän · ‎09-26-2017

We are attempting to use XL710 together with Openstack, and acceleration using SR-IOV.

With the base drivers provided by Ubuntu 14, VM's can be created but during boot cloud-init fails to create any interfaces with the following error message:

[ 7.562357] cloud-init[395]: ret = functor(name, args)

[ 7.563806] cloud-init[395]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 364, in main_init

[ 7.568756] cloud-init[395]: init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))

[ 7.626863] cloud-init[395]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 635, in apply_network_config

[ 7.627928] cloud-init[395]: netcfg, src = self._find_networking_config()

[ 7.628708] cloud-init[395]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 622, in _find_networking_config

[ 7.632059] cloud-init[395]: if self.datasource and hasattr(self.datasource, 'network_config'):

[ 7.636062] cloud-init[395]: File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py", line 150, in network_config

[ 7.644072] cloud-init[395]: self.network_json, known_macs=self.known_macs)

[ 7.705553] cloud-init[395]: File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/openstack.py", line 652, in convert_net_json

[ 7.706582] cloud-init[395]: raise ValueError("Unable to find a system nic for %s" % d)

[ 7.707395] cloud-init[395]: ValueError: Unable to find a system nic for {'mtu': 1500, 'type': 'physical', 'mac_address': 'fa:16:3e:c8:81:d9', 'subnets': [{'routes': [], 'type': 'static', 'address': '10.0.20.6', 'netmask': '255.255.255.0', 'ipv4': True}]}

When installing the latest drivers available from Intel, VM's can not be created at all. Looking at the OpenStack logs, this error message can be found: "internal error: couldn't find IFLA_VF_INFO for VF 7 in netlink response". Searching for this error on google it seems like there could be problems when creating more than 30 VF's in some older versions of libvirt, but we are only testing with 8 VF's right now.

We are running Ubuntu 14 and Openstack Mitaka. We have tried a later Ubuntu/Openstack version (Ubuntu 16 and OpenStack Newton) but there were problems there too -- unsure if it was the same issue, since the logs are long gone.

We have the latest drivers and firmware updates installed. We do -not- have the i40evf driver installed since that would conflict with the VM claiming VF's according to this document https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf .

When configuring SR-IOV we are following the guide on https://docs.openstack.org/mitaka/networking-guide/config-sriov.html https://docs.openstack.org/mitaka/networking-guide/config-sriov.html

We have some other Intel cards (I can't remember the model) which this works fine for.

idata · ‎09-26-2017

Hi TomasBrn,

Thank you for posting in Wired Communities. Please provide below information:

1) What is the firmware of your XL710?

2) Please provide the serial number, you may refer to this website for the serial number location on the physical NIC

https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007022.html

The format is 15 digits + 6 digits + 6 digits - 3 digits

Thanks,

Sharon

TBrän · ‎09-27-2017

Hi

Thanks for the reply.

More specifically we have XL710-QDA2 cards.

The firmware is the latest from the Intel website, from ethtool in Linux:

root # ethtool -i ens2f0

driver: i40e

version: 1.5.19

firmware-version: 6.01 0x800034a4 1.1747.0

The serial numbers for the cards are:

3CFDFEAFDB48

008 3317AD

J11363-003

3CFDFEAFE2F0

008 3317AD

J11363-003

3CFDFEAFECF0

008 3317AD

J11363-003

/Tomas

idata · ‎09-27-2017

Hi TomasBrn,

Thank you for the information. I will further check on this.

Regards,

Sharon

idata · ‎10-01-2017

Hi TomasBrn,

Please help verify if you updated to the latest driver version at website below:

https://sourceforge.net/projects/e1000/files/i40e%20stable/

Looking forward to your reply. Thank you.

Regards,

Sharon

TBrän · ‎10-02-2017

Yes we were running the latest drivers (2.1.26). (the ethtool output above was taken after an OS reinstall, before drivers were installed)

/Tomas

idata · ‎10-02-2017

Hi TomasBrn,

Thank you for the confirmation. Will further check for you.

Regards,

Sharon

TBrän · ‎10-05-2017

Hi

Any further progress?

I can mention that our working 10Gb cards uses the "ixgbe" drivers, and in the VM the virtual functions get the "ixgbevf" driver. They are otherwise configured in the same way as the XL710-QDA2 cards

EDIT: I just did some more testing and noticed that the i40evf driver was actually loaded, which probably was why we got the "internal error: couldn't find IFLA_VF_INFO for VF 7 in netlink response" error. We had missed to blacklist it on the server it seems. But with the module blacklisted, we still get this error from cloud-init (same as the original post with the older drivers):

[ 7.818035] cloud-init[791]: Traceback (most recent call last):

[ 7.818915] cloud-init[791]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 650, in status_wrapper

[ 7.820120] cloud-init[791]: ret = functor(name, args)

[ 7.820960] cloud-init[791]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 364, in main_init

[ 7.822095] cloud-init[791]: init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))

[ 7.823158] cloud-init[791]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 635, in apply_network_config

[ 7.824420] cloud-init[791]: netcfg, src = self._find_networking_config()

[ 7.825430] cloud-init[791]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 622, in _find_networking_config

[ 7.826657] cloud-init[791]: if self.datasource and hasattr(self.datasource, 'network_config'):

[ 7.827754] cloud-init[791]: File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py", line 150, in network_config

[ 7.829097] cloud-init[791]: self.network_json, known_macs=self.known_macs)

[ 7.830092] cloud-init[791]: File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/openstack.py", line 652, in convert_net_json

[ 7.831434] cloud-init[791]: raise ValueError("Unable to find a system nic for %s" % d)

[ 7.832494] cloud-init[791]: ValueError: Unable to find a system nic for {'mac_address': 'fa:16:3e:3e:d1:dd', 'mtu': 1500, 'subnets': [{'ipv4': True, 'routes': [], 'netmask': '255.255.255.0', 'address': '10.0.30.100', 'type': 'static'}], 'type': 'physical'}

EDIT2: Actually, I realized that the document https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf https://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief… mentions that RHEL is a prerequisite. So I tried to create a centos based VM, and it can successfully boot and seems to have a working interface! I have not verified connectivity but it has successfully loaded the drivers etc.

So I guess my question changes a bit: Is there actually no support for SR-IOV for this card for Ubuntu based cloud virtual machines?

/Tomas

idata · ‎10-08-2017

Hi TomasBrn,

Thank you for the information, I was about to ask you to provide more information such as

Linux Kernel log from Ubuntu host that is having the issue. With the new setup you have, I will further check for you about your new inquiry.

Regards,

Sharon

idata · ‎10-11-2017

Hi TomasBrn,

Please provide below information for better investigation:

1) Host OS and version

2) VM OS and version

3) Is the Ubuntu problem happening only on the VM?

4) What combinations fail and what work?

Thanks,

Sharon

idata · ‎10-16-2017

Hi TomasBrn,

Please feel free to provide the information requested on my previous post. Thank you.

Thanks,

Sharon

TBrän · ‎10-16-2017

Sorry, I missed the last post

* The host OS is Ubuntu 14.04.5 server version

* The VM OS is Ubuntu 16.04 server based cloud image https://cloud-images.ubuntu.com/xenial/current/ Ubuntu 16.04 LTS (Xenial Xerus) Daily Build [20171011]

* Yes, we only see it in the VM. So I suspect the cloud image does not have the drivers.

* What we tried that worked was using CentOS based cloud image instead.

/Tomas

idata · ‎10-17-2017

HI TomasBrn,

Thank you for the information, I will further check and update you once there is any finding.

Regards,

Sharon

idata · ‎10-19-2017

Hi TomasBrn,

Understand that there is no issue using Centos cloud image. Further checking, we are not able to comment on why Ubuntu cloud image fails since there are various possibilities such as image issue, driver problem or other factors.

Regards,

Sharon

TBrän · ‎10-24-2017

OK. Since it works on centos we will have to use that for now. Thank you for your efforts

/Tomas

idata · ‎10-24-2017

Hi TomasBrn,

You are welcome and appreciate your time on this matter.

Regards,

Sharon