X710-DA4 - SR-IOV not working correctly

[DMarq1]

Host OS: Ubuntu 16.04 LTS

Hypervisor: Xen 4.6.0

Kernel version: 4.4.0-22-generic

Driver version: i40e-1.5.18

NVM version: 5.02

Guest OS: Windows Server 2012 R2

Guest Driver version: 20.7.1

I'm trying to forward the ports of the card to a guest Windows VM. An example of forwarding just one port:

# echo 1 > /sys/class/net/ens2f0/device/sriov_numvfs

# /sbin/ip link set ens2f0 vf 0 mac 02:00:00:00:00:00

# xl pci-assignable-add 02:02.0

Added to Xen VM .cfg:

pci = ['0000:02:02.0']

All of that works and the VM starts fine and sees the card. Xen reports that it added it:

[26326.586765] i40e 0000:02:00.0 ens2f0: adding 68:05:ca:33:0a:59 vid=0

[26326.586774] i40e 0000:02:00.0 ens2f0: adding 68:05:ca:33:0a:59 vid=1

[26326.595995] i40e 0000:02:00.0: Allocating 1 VFs.

[26326.697778] pci 0000:02:02.0: [8086:154c] type 00 class 0x020000

[26326.697973] pci 0000:02:02.0: Max Payload Size set to 256 (was 128, max 2048)

[26329.893582] i40e 0000:02:00.0: Setting MAC 02:00:00:00:00:00 on VF 0

[26329.965339] i40e 0000:02:00.0: Reload the VF driver to make this change effective.

[26366.200750] pciback 0000:02:02.0: seizing device

[26366.200815] pciback 0000:02:02.0: enabling device (0000 -> 0002)

[26472.867101] xen_pciback: vpci: 0000:02:02.0: assign to virtual slot 0

[26472.867643] pciback 0000:02:02.0: registering for 4

I install the drivers and Windows 'see' the card. The problem is the reported state goes into a cycle:

Connected @ 10GB

Connected @ 40GB

Disconnected

When this happens, in dmesg on the host:

[26624.137415] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26635.075640] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26645.903762] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26656.669642] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26667.904191] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26679.013381] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26689.966331] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26701.028906] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26712.372668] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26723.154159] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26733.935320] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26744.685130] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

[26756.075844] i40e 0000:02:00.0: VF 0 failed opcode 11, retval: -10

This same issue happens using the built in kernel i40e and also i40e-1.5.18. Am I doing something wrong?

[idata]

Hi,

Thank you for posting your concern in Intel Wired Ethernet communities.You might need to unload the VF driver from the host OS, kindly refer to page 14 of this article - http://www.intel.com/content/www/us/en/embedded/products/networking/xl710-sr-iov-config-guide-gbe-linux-brief.htmlAdditional reference related to SR-IOV implementation: http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.htmlRegards,Vince

[DMarq1]

VF driver already blacklisted:

# cat /lib/modprobe.d/vm-blacklist.conf

# Don't try and use VFs so that we can pass them through

blacklist i40evf

# lsmod | grep i40

i40e 319488 0

vxlan 49152 1 i40e

ptp 20480 2 igb,i40e

[DMarq1]

BTW, I was following:

http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.pdf http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/xl710-sr-iov-config-guide-gbe-linux-brief.…

[DMarq1]

I notice that NVM 5.02 has gotten pulled. Could this be an issue caused by 5.02?

[idata]

Hi,

The NVM files have been temporarily removed and we will post the updates as soon as we can. For your issue, i'm still checking on it. Regards,Vince

[idata]

Hi Jailbird, while we're still checking your concern, please share the product label description found of your X710-DA4. This information is visible in the white sticker found on the physical adapter.

Kindly refer to sample label from the link below: http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007060.htmlregards,Vince

[idata]

Hi Jailbird, we'd like to follow-up the product label description of your X710-DA4. Thank you.

Regards,Vince

[DMarq1]

The sticker on the front has:

008 5214AD

H58361-002

The sticker on the back has:

INTEL(R) ETHERNET CNA X710-DA4 FH

932576

X710DA4FHBLK

Date of Manufacture

12/2014

Made in Malaysia

[idata]

Hi Jailbird,

Thank you for the information. we will further check on your inquiry,rgds,wb

[idata]

Hi Jailbird,

Good day. There is a newer i40e driver available at sourceforge.please try install and check: https://sourceforge.net/projects/e1000/files/i40e%20stable/ Also, could provide the output of the 'ethtool -i' command for the specific XL710 port in question and any kernel logs? Please update if you have tried the same NIC in a non-virtualized environment running the same guest OS and versions? Thanks,wb

[idata]

HI Jailbird,

Please feel free to update the test result using the driver provided on my previous postand the additional information.thanks,wb