SR-IOV not enabling on a X710's second port

[vlsnt]

Hello,

On a Dell R640 equipped with a I350/X710 network daughter card, SR-IOV doesn't appear to be enabled on the second X710 port, despite the fact it is in "System Setup" > "Device Settings" and globally, in "Integrated Devices".

The host is running RedHat Linux 8.0.
The network card's firmware version is 19.5.12.

What could be the cause ? Can it be a hardware defect ?

Thank you in advance.

[root@xxx ~]# cd /sys/class/net/eno2/device/
[root@xxx device]# ls -la | grep sr
[root@xxx device]# cd /sys/class/net/eno1/device/
[root@xxx device]# ls -la | grep sr
-rw-rw-r-- 1 root root 4096 Oct 5 14:56 sriov_drivers_autoprobe
-rw-rw-r-- 1 root root 4096 Oct 5 14:56 sriov_numvfs
-r--r--r-- 1 root root 4096 Oct 5 14:56 sriov_offset
-r--r--r-- 1 root root 4096 Oct 5 14:56 sriov_stride
-r--r--r-- 1 root root 4096 Oct 5 14:56 sriov_totalvfs
-r--r--r-- 1 root root 4096 Oct 5 14:56 sriov_vf_device
[root@xxx device]#

*-network:0
description: Ethernet interface
product: Ethernet Controller X710 for 10GbE SFP+
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:19:00.0
logical name: eno1
version: 02
serial: e4:43:4b:a6:31:76
size: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre autonegotiation
configuration: autonegotiation=off broadcast=yes driver=i40e driverversion=2.3.2-k duplex=full firmware=7.10 0x800075da 19.5.12 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
resources: irq:103 memory:9f000000-9fffffff memory:a0408000-a040ffff memory:9d800000-9d87ffff memory:a0000000-a03fffff memory:a0410000-a050ffff
*-network:1
description: Ethernet interface
product: Ethernet Controller X710 for 10GbE SFP+
vendor: Intel Corporation
physical id: 0.1
bus info: pci@0000:19:00.1
logical name: eno2
version: 02
serial: e4:43:4b:a6:31:78
size: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre autonegotiation
configuration: autonegotiation=off broadcast=yes driver=i40e driverversion=2.3.2-k duplex=full firmware=7.10 0x800075da 19.5.12 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
resources: irq:103 memory:9e000000-9effffff memory:a0400000-a0407fff memory:9d880000-9d8fffff

[vlsnt]

Hello,

Here is some additional data. The kernel has been upgraded (and brought a new driver version), however this didn't change the problem. Note the "VFs: 0" exclusively on device 0000:19:00.1...

$ dmesg | grep i40
[ 4.020209] i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.8.20-k
[ 4.020748] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[ 4.065048] i40e 0000:19:00.0: fw 7.83.59945 api 1.9 nvm 7.10 0x800075da 19.5.12 [8086:1572] [1028:1f99]
[ 4.310015] i40e 0000:19:00.0: MAC address: e4:43:4b:a6:31:76
[ 4.310805] i40e 0000:19:00.0: FW LLDP is enabled
[ 4.359591] i40e 0000:19:00.0: PCI-Express: Speed 8.0GT/s Width x8
[ 4.368639] i40e 0000:19:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 4.677640] i40e 0000:19:00.1: fw 7.83.59945 api 1.9 nvm 7.10 0x800075da 19.5.12 [8086:1572] [1028:0000]
[ 4.926579] i40e 0000:19:00.1: MAC address: e4:43:4b:a6:31:78
[ 4.926713] i40e 0000:19:00.1: FW LLDP is enabled
[ 4.937104] i40e 0000:19:00.1 eth2: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 4.949191] i40e 0000:19:00.1: PCI-Express: Speed 8.0GT/s Width x8
[ 4.977677] i40e 0000:19:00.1: Features: PF-id[1] VFs: 0 VSIs: 2 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 4.992289] i40e 0000:d8:00.0: fw 7.83.59945 api 1.9 nvm 7.10 0x800075df 19.5.12 [8086:1572] [8086:0006]
[ 5.261425] i40e 0000:d8:00.0: MAC address: f8:f2:1e:b3:06:90
[ 5.261949] i40e 0000:d8:00.0: FW LLDP is enabled
[ 5.289930] i40e 0000:d8:00.0: PCI-Express: Speed 8.0GT/s Width x8
[ 5.299407] i40e 0000:d8:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 5.312714] i40e 0000:d8:00.1: fw 7.83.59945 api 1.9 nvm 7.10 0x800075df 19.5.12 [8086:1572] [8086:0000]
[ 5.544321] i40e 0000:d8:00.1: MAC address: f8:f2:1e:b3:06:91
[ 5.544949] i40e 0000:d8:00.1: FW LLDP is enabled
[ 5.566874] i40e 0000:d8:00.1: PCI-Express: Speed 8.0GT/s Width x8
[ 5.576389] i40e 0000:d8:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 5.578739] i40e 0000:19:00.0 eno1: renamed from eth0
[ 5.590339] i40e 0000:19:00.1 eno2: renamed from eth2
[ 5.600375] i40e 0000:d8:00.0 ens3f0: renamed from eth1
[ 5.614259] i40e 0000:d8:00.1 ens3f1: renamed from eth3
[ 39.525974] i40e 0000:19:00.0 eno1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None

[Caguicla_Intel]

Hello vlsnt,

Thank you for posting in Intel Ethernet Communities. 

Please provide the following information that would help in checking your request.

1.) Share the link of the firmware where it was downloaded from. 

2.) Please share the link where the i40e and iavf drivers were downloaded. 

3.) Kindly provide the PBA and serial number of the adapter. You may refer to the link below on where to find the PBA number. Providing photos of the adapter focusing on the markings (white sticker) found on the physical card will be highly appreciated for us to double check on it. The PBA consists 6-3 digit number located at the last part of the serial number.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html

4.) Kindly confirm if the steps on the SR-IOV Configuration Guide link below were followed. 

https://www.intel.com/content/dam/www/public/us/en/documents/guides/sr-iov-guide-ethernet-x710-xl710-linux-brief.pdf

Looking forward to your reply. 

Should there be no response, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C

Intel® Customer Support

[vlsnt]

Hello Crisselle,

Thank your for your interrest in that matter.

1- The server shipped from factory with 19.5.12 already present on the NIC. You can find the Dell package for that NIC at this  location: https://www.dell.com/support/home/fr-fr/drivers/driversdetails?driverid=yp4r0&oscode=wst14&productcode=poweredge-r640

2- The drivers are those from RedHat Enterprise Linux 8, kernel 4.18.0-193.19.1.el8_2.x86_64.

3- I'll ask our field engineer to take a picture.

4- On page 9 of this document, § 12.b., we are instructed to configure the number of VFs by writing to file /sys/class/net/device name/device/sriov_numvfs. That's precisely the problem: this file does not exist, specifically for the second port of the integrated X710. While it does exist for the first port. Probably this is linked to the fact the kernel driver displays "VFs: 0" for the second port, when it is loaded (while it displays 64 for the first port).

Thank you.

[Caguicla_Intel]

Hello vlsnt,

Thank you for the reply.

We will patiently wait for the photos of the adapter. Please be informed that if the adapter turns out to be an Original equipment Manufacturer (OEM) version, we might refer you to the manufacturer for further assistance. As for OEM adapter, it would be best if you can also get in touch with the manufacturer for further support since they have customized drivers and software to enable or alter features on your card. They are in best position where you can seek assistance so you don’t lose features or customizations of the Ethernet Adapter.

We'd also like to check the driver version of Virtual function (iavf) used as well as the link where it was downloaded from. Additionally, please let us know if you already tried the latest driver for your adapter and share the result with us.

Hoping to hear from you. 

We'll follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C

Intel® Customer Support

[RUrena]

I had a case very similar to this one and it seems the fw is 7.10. In my case the driver used is the one included with RHEL 8.0. I also saw another case [1] that was not RHEL related that had a proposed solution of upgrading the firmware. In that case it seems that the second port was using only 1 VF. In my case i see the below:

# dmesg | grep i40e | grep VF
[ 6.016843] i40e 0000:19:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 6.898113] i40e 0000:19:00.1: Features: PF-id[1] VFs: 0 VSIs: 2 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 7.481595] i40e 0000:d8:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 8.078595] i40e 0000:d8:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 48 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA

Note that all the other ports get the VFs: 64. i am also curious the documentation states that the x710 should only have 32 VFs per port but i see 64. I am not sure about the last 2 when i checked they were using the igb driver and not the i40e driver.  I've asked the customer to update to the 8.10 firmware to see if that solves the problem. I would like it if you guys could see if this problem is related to the one i found. This one is closer to the one i have.

[1] https://community.intel.com/t5/Ethernet-Products/Issues-with-second-physical-interface-on-2P-X710-using-SR-IOV/m-p/728909#M20053%3Fwapkw=x710%20sr-iov

[vlsnt]

Hello @Caguicla_Intel ,

I apologize for the delay in getting back to you. Would you please accept to resume investigations on this case ?

I managed to obtain the adapter's PBA via the server's system setup utility: K15075-001
Model name: Intel Ethernet 10G 4P X710/I350

Some additional data :

# ethtool -i eno2
driver: i40e
version: 2.8.20-k
firmware-version: 7.10 0x800075da 19.5.12
expansion-rom-version:
bus-info: 0000:19:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Also note that Dell replaced both the motherboard and this integrated NIC... the problem persists.

Thank you in advance.

[Caguicla_Intel]

Hello vlsnt,

Good day!

We'd like to follow up the additional information that we requested. In case you need more time in gathering the details, feel free to let us know.

Awaiting to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello vlsnt,

How are you doing? 

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. If you need any additional information, please submit a new question as this thread will no longer being monitored.

May you have an amazing day ahead!

Best regards,

Crisselle C

Intel® Customer Support