Intel SR-IOV XL710 VF doesn't work after VM reboot

[KPoku]

I'm using Ubuntu 16.04.3 LTS with KVM virtualization. Also i'm using

Intel XL710 card with SR-IOV feature.

Here are the outputs:

/lib/modprobe.d$ modinfo i40e

filename:      /lib/modules/4.4.0-96-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko

version:       2.4.10

license:       GPL

description:   Intel(R) 40-10 Gigabit Ethernet Connection Network Driver

author:        Intel Corporation, <e1000-devel@lists.sourceforge.net>

srcversion:    3977C21019A3C4865FF253A

alias:         pci:v00008086d0000158Bsv*sd*bc*sc*i*

alias:         pci:v00008086d0000158Asv*sd*bc*sc*i*

alias:         pci:v00008086d000037D3sv*sd*bc*sc*i*

alias:         pci:v00008086d000037D2sv*sd*bc*sc*i*

alias:         pci:v00008086d000037D1sv*sd*bc*sc*i*

alias:         pci:v00008086d000037D0sv*sd*bc*sc*i*

alias:         pci:v00008086d000037CFsv*sd*bc*sc*i*

alias:         pci:v00008086d000037CEsv*sd*bc*sc*i*

alias:         pci:v00008086d00001588sv*sd*bc*sc*i*

alias:         pci:v00008086d00001587sv*sd*bc*sc*i*

alias:         pci:v00008086d00001589sv*sd*bc*sc*i*

alias:         pci:v00008086d00001586sv*sd*bc*sc*i*

alias:         pci:v00008086d00001585sv*sd*bc*sc*i*

alias:         pci:v00008086d00001584sv*sd*bc*sc*i*

alias:         pci:v00008086d00001583sv*sd*bc*sc*i*

alias:         pci:v00008086d00001581sv*sd*bc*sc*i*

alias:         pci:v00008086d00001580sv*sd*bc*sc*i*

alias:         pci:v00008086d00001574sv*sd*bc*sc*i*

alias:         pci:v00008086d00001572sv*sd*bc*sc*i*

depends:       ptp,vxlan

vermagic:      4.4.0-96-generic SMP mod_unload modversions

parm:          debug:Debug level (0=none,...,16=all) (int)

 cat /etc/rc.local

#!/bin/sh -e

#

# rc.local

#

#

# By default this script does nothing.

echo 8 > /sys/class/net/enp6s0f1/device/sriov_numvfs

echo 8 > /sys/class/net/enp6s0f0/device/sriov_numvfs

ip link set enp6s0f0 vf 0 mac aa:bb:cc:aa:00:02 vlan 12

ip link set enp6s0f0 vf 1 mac aa:bb:cc:aa:01:02 vlan 32

ip link set enp6s0f0 vf 2 mac aa:ff:ff:aa:00:12 vlan 22

ip link set enp6s0f0 vf 3 mac aa:ff:ff:aa:01:12 vlan 42

ip link set enp6s0f1 vf 0 mac aa:bb:cc:bb:00:02 vlan 52

ip link set enp6s0f1 vf 1 mac aa:bb:cc:bb:01:02 vlan 72

ip link set enp6s0f1 vf 2 mac aa:ff:ff:bb:00:12 vlan 62

ip link set enp6s0f1 vf 3 mac aa:ff:ff:bb:01:12 vlan 82

exit 0

Driver i40e is also blacklisted as suggested in Intel SR-IOV configuration guide.

Everything works ok, but in case i reboot the VM, some of the virtual functions won't work. I need to reboot VM a few times (sometimes i need just one reboot, sometimes like three reboots) and virtual functions start working properly.

I was able only to find this in logs on KVM host (/var/log/kern.log) when VF doesn't work:

Jan 8 09:56:23 ri-cgn-kvm-2 kernel: [15787510.840948] i40e 0000:06:00.0: VF 1 failed opcode 3, retval: -5

Jan 8 09:56:23 ri-cgn-kvm-2 kernel: [15787510.841010] i40e 0000:06:00.0: Invalid message from VF 1, opcode 3, len 4

Jan 8 09:57:08 ri-cgn-kvm-2 kernel: [15787555.936159] i40e 0000:06:00.1: VF 0 failed opcode 10, retval: -5

Jan 8 09:57:08 ri-cgn-kvm-2 kernel: [15787555.936222] i40e 0000:06:00.1: Invalid message from VF 0, opcode 10, len 44

Jan 8 09:57:08 ri-cgn-kvm-2 kernel: [15787555.976198] i40e 0000:06:00.1: VF 0 failed opcode 11, retval: -10

Jan 8 10:05:14 ri-cgn-kvm-2 kernel: [15788042.047074] i40e 0000:06:00.0: VF 0 failed opcode 3, retval: -5

Jan 8 10:05:14 ri-cgn-kvm-2 kernel: [15788042.047138] i40e 0000:06:00.0: Invalid message from VF 0, opcode 3, len 4

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.950375] i40e 0000:06:00.0: VF is not trusted, switch the VF to trusted to add more functionality

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.951291] i40e 0000:06:00.1: VF is not trusted, switch the VF to trusted to add more functionality

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.951300] i40e 0000:06:00.1: VF 1 failed opcode 10, retval: -1

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.977115] i40e 0000:06:00.0: VF 1 failed opcode 10, retval: -1

Jan 8 10:37:25 ri-cgn-kvm-2 kernel: [15789972.835652] i40e 0000:06:00.0: VF 1 failed opcode 11, retval: -10

Jan 8 10:37:27 ri-cgn-kvm-2 kernel: [15789975.074702] i40e 0000:06:00.1: VF 1 failed opcode 11, retval: -10

What is causing this issue?

Regards,

Kresimir

[Daniel_D_Intel1]

Hello KPoku, Thank you for posting in Intel Ethernet Communities. Please let us know if this is a retail Intel version of the XL710 adapter or an OEM by providing the markings from the product label. Please include ethtool -i <interface> output to confirm the firmware, and ethtool -k <interface> to provide the features on the NIC. Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support

[KPoku]

Hi Daniel,

here are outputs of ethtool command:

ethtool -i enp1s0f0

driver: igb

version: 5.3.0-k

firmware-version: 1.63, 0x80000e74, 0.385.8

expansion-rom-version:

bus-info: 0000:01:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

ethtool -k enp1s0f0

Features for enp1s0f0:

rx-checksumming: on

tx-checksumming: on

       tx-checksum-ipv4: on

       tx-checksum-ip-generic: off [fixed]

       tx-checksum-ipv6: on

       tx-checksum-fcoe-crc: off [fixed]

       tx-checksum-sctp: on

scatter-gather: on

       tx-scatter-gather: on

       tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

       tx-tcp-segmentation: on

       tx-tcp-ecn-segmentation: off [fixed]

       tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

busy-poll: off [fixed]

hw-tc-offload: off [fixed]

ethtool -i enp1s0f1

driver: igb

version: 5.3.0-k

firmware-version: 1.63, 0x80000e74, 0.385.8

expansion-rom-version:

bus-info: 0000:01:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

ethtool -k enp1s0f1

Features for enp1s0f1:

rx-checksumming: on

tx-checksumming: on

       tx-checksum-ipv4: on

       tx-checksum-ip-generic: off [fixed]

       tx-checksum-ipv6: on

       tx-checksum-fcoe-crc: off [fixed]

       tx-checksum-sctp: on

scatter-gather: on

       tx-scatter-gather: on

       tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

       tx-tcp-segmentation: on

       tx-tcp-ecn-segmentation: off [fixed]

       tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

busy-poll: off [fixed]

hw-tc-offload: off [fixed]

This Intel cards were ordered and provided with Cisco. I'm running CISCO UCS C220M4S server.

Regards,

Kresimir

[Daniel_D_Intel1 · Accepted solution]

Hello KPoku, Thank you for the information. The kernel is not providing the latest i40e drivers. Please check if updating to the latest i40e drivers resolves the issue. Please check with Cisco if there is a firmware update available for the XL710, as this is an OEM Cisco version. They might also be able to provide the compatibility of drivers with the latest firmware. You can find the latest drivers here: https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux- Let us know if the drivers and firmware update solve this issue. If you have any other questions please inform us. Best regards, Daniel D Intel Customer Support

[Daniel_D_Intel1]

Hello KPoku, Please let us know if you are still having any issues after updating to the latest i40e driver and checking if there is a firmware update from Cisco. If you have any other questions please do not hesitate to ask. Best regards, Daniel D Intel Customer Support

[Daniel_D_Intel1]

Hello KPoku, Please let us know if you still need assistance. If you have any other questions please do not hesitate to ask. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[KPoku]

Hi Daneil,

i've installed latest drivers, v.2.7.26, one host and didn't experience issue with VF.

I'll be testing for next couple of days to see will issue appear.

Regards,

Kresimir

[Daniel_D_Intel1]

Hello KPoku, Thank you for the reply. Please let us know if the issue is still seen in your testing. If you have any other questions please ask. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[Daniel_D_Intel1]

Hello KPoku, After testing the latest driver for some time are you still getting the expected VF performance? Please let us know if you need additional assistance, or have any questions. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[Daniel_D_Intel1]

Hello KPoku, Please let us know if you have any questions. If the driver update solved the issue please inform us. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[Mike_Intel]

Hello KPoku, Please tell us If the driver update solved the issue. Kindly tell us if you have any questions. Best regards, Michael L. Intel Customer Support Under contract to Intel Corporation

[KPoku]

Hi Michael,

for now i didn't have any issues with latest drivers, so i can say that issue is solved.

Thank you for the assistance.

Regards,

Kresimir