topic Re: X710-DA4 - SR-IOV not working correctly in Ethernet Products

X710-DA4 - SR-IOV not working correctly

DMarq1 — Fri, 13 May 2016 00:40:02 GMT

Host OS: Ubuntu 16.04 LTS

Hypervisor: Xen 4.6.0

Kernel version: 4.4.0-22-generic

Driver version: i40e-1.5.18

NVM version: 5.02

Guest OS: Windows Server 2012 R2

Guest Driver version: 20.7.1

I'm trying to forward the ports of the card to a guest Windows VM. An example of forwarding just one port:

# echo 1 > /sys/class/net/ens2f0/device/sriov_numvfs

# /sbin/ip link set ens2f0 vf 0 mac 02:00:00:00:00:00

# xl pci-assignable-add 02:02.0

Added to Xen VM .cfg:

pci = ['0000:02:02.0']

All of that works and the VM starts fine and sees the card. Xen reports that it added it:

[26326.586765] i40e 0000:02:00.0 ens2f0: adding 68:05:ca:33:0a:59 vid=0

[26326.586774] i40e 0000:02:00.0 ens2f0: adding 68:05:ca:33:0a:59 vid=1

[26326.595995] i40e 0000:02:00.0: Allocating 1 VFs.

[26326.697778] pci 0000:02:02.0: [8086:154c] type 00 class 0x020000

[26326.697973] pci 0000:02:02.0: Max Payload Size set to 256 (was 128, max 2048)

[26329.893582] i40e 0000:02:00.0: Setting MAC 02:00:00:00:00:00 on VF 0

[26329.965339] i40e 0000:02:00.0: Reload the VF driver to make this change effective.

[26366.200750] pciback 0000:02:02.0: seizing device

[26366.200815] pciback 0000:02:02.0: enabling device (0000 -> 0002)

[26472.867101] xen_pciback: vpci: 0000:02:02.0: assign to virtual slot 0

[26472.867643] pciback 0000:02:02.0: registering for 4

I install the drivers and Windows 'see' the card. The problem is the reported state goes into a cycle:

Connected @ 10GB

Connected @ 40GB

Disconnected

When this happens, in dmesg on the host:

[26624.137415] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26635.075640] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26645.903762] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26656.669642] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26667.904191] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26679.013381] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26689.966331] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26701.028906] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26712.372668] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26723.154159] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26733.935320] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26744.685130] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26756.075844] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

This same issue happens using the built in kernel i40e and also i40e-1.5.18. Am I doing something wrong?

Re: X710-DA4 - SR-IOV not working correctly

idata — Wed, 18 May 2016 08:41:45 GMT

Hi,

Thank you for posting your concern in Intel Wired Ethernet communities.

You might need to unload the VF driver from the host OS, kindly refer to page 14 of this article - http://www.intel.com/content/www/us/en/embedded/products/networking/xl710-sr-iov-config-guide-gbe-linux-brief.html

Additional reference related to SR-IOV implementation:

http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.html

Regards,

Vince

Re: X710-DA4 - SR-IOV not working correctly

DMarq1 — Wed, 18 May 2016 23:09:47 GMT

VF driver already blacklisted:

# cat /lib/modprobe.d/vm-blacklist.conf

# Don't try and use VFs so that we can pass them through

blacklist i40evf

# lsmod | grep i40

i40e 319488 0

vxlan 49152 1 i40e

ptp 20480 2 igb,i40e

Re: X710-DA4 - SR-IOV not working correctly

DMarq1 — Wed, 18 May 2016 23:18:40 GMT

BTW, I was following:

http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.…

Re: X710-DA4 - SR-IOV not working correctly

DMarq1 — Thu, 19 May 2016 06:27:57 GMT

I notice that NVM 5.02 has gotten pulled. Could this be an issue caused by 5.02?

Re: X710-DA4 - SR-IOV not working correctly

idata — Tue, 24 May 2016 10:42:31 GMT

Hi,

The NVM files have been temporarily removed and we will post the updates as soon as we can. For your issue, i'm still checking on it.

Regards,

Vince

Re: X710-DA4 - SR-IOV not working correctly

idata — Fri, 27 May 2016 10:10:20 GMT

Hi Jailbird, while we're still checking your concern, please share the product label description found of your X710-DA4. This information is visible in the white sticker found on the physical adapter.

Kindly refer to sample label from the link below:

http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007060.html

regards,

Vince

Re: X710-DA4 - SR-IOV not working correctly

idata — Fri, 03 Jun 2016 10:46:49 GMT

Hi Jailbird, we'd like to follow-up the product label description of your X710-DA4. Thank you.

Regards,

Vince

Re: X710-DA4 - SR-IOV not working correctly

DMarq1 — Fri, 17 Jun 2016 21:55:28 GMT

The sticker on the front has:

008 5214AD

H58361-002

The sticker on the back has:

INTEL(R) ETHERNET CNA X710-DA4 FH

932576

X710DA4FHBLK

Date of Manufacture

12/2014

Made in Malaysia

Re: X710-DA4 - SR-IOV not working correctly

idata — Tue, 21 Jun 2016 01:27:21 GMT

Hi Jailbird,

Thank you for the information. we will further check on your inquiry,

rgds,

Re: X710-DA4 - SR-IOV not working correctly

idata — Thu, 18 Aug 2016 04:26:49 GMT

Hi Jailbird,

Good day. There is a newer i40e driver available at sourceforge.please try install and check:

https://sourceforge.net/projects/e1000/files/i40e%20stable/

Also, could provide the output of the 'ethtool -i' command for the specific XL710 port in question and any kernel logs?

Please update if you have tried the same NIC in a non-virtualized environment running the same guest OS and versions?

Thanks,

Re: X710-DA4 - SR-IOV not working correctly

idata — Thu, 25 Aug 2016 01:48:05 GMT

HI Jailbird,

Please feel free to update the test result using the driver provided on my previous post

and the additional information.

thanks,