Re: Ubuntu 16.04 and Intel XL710 SR-IOV - Packet drops - Intel Community

Ethernet Products

Determine ramifications of Intel® Ethernet products and technologies

Ubuntu 16.04 and Intel XL710 SR-IOV - Packet drops

5,236 Views

We have server running Ubuntu 16.4 KVM with Intel XL710 40Gbps NIC with SR-IOV on top of it.

On top of each 40G interfaces are created 4 SR-IOV VFs which are associated to VMs.

Problem is that after traffic load increases (approx 6Gbps on physical intf) we are experencing increased RTT and traffic drops.

VMs looks OK, switches and rest of the network also looks good and we are suspecting to SR-IOV.

How can I verify potential traffic drops on SR-IOV?

Has anybody similar expirience?

Kernel: 4.4.0-66-generic

driver=i40e driverversion=1.4.25-k duplex=full firmware=5.04

BR, Mate

Link Copied

14 Replies

3,662 Views

Hi Mate,

Please try upgrading the NVM firmware to version 5.0.5

https://downloadcenter.intel.com/download/25791/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Adapters-Linux-?product=83967

Understand you are using Ubuntu OS, the driver we have is for Linux*, version 1.6.4.2

https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-Intel-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=83967

Virtual function driver version 1.6.41

https://downloadcenter.intel.com/download/24693/Intel-Network-Adapter-Virtual-Function-Driver-for-Intel-40-Gigabit-Ethernet-Network-Connections?product=83967

Please try check if the driver is applicable.

Thanks,

wb

Copy link

3,662 Views

Hi,

tnx. I think we already tried this, but we had some problems with loading drivers. We'll try again and post results here.

Meanwhile, how can we check is there any packet drops/errors on VF and PF?

ethtool -S enp6s0f0 shows no errors or drops

BR, Mate

Copy link

3,662 Views

Hi Mate,

Thank you for the reply, please feel free to update me the result.

I will check on this.

rgds,

wb

Copy link

3,662 Views

Hi,

in first post I forgot to write down full NIC model, it is Intel XL710-QDA2.

We tried to update FW and got following error/status (Update not available):

:

./nvmupdate64e

Intel(R) Ethernet NVM Update Tool

NVMUpdate version 1.28.19.4

Copyright (C) 2013 - 2016 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

Inventory in progress. Please wait [|.........]

Num Description Ver. DevId S:B Status

=== ======================================== ===== ===== ====== ===============

01) Intel(R) I350 Gigabit Network Connection 1.99 1521 00:001 Update not

available

02) Cisco(R) Ethernet Converged NIC 5.04 1583 00:006 Update not

XL710-QDA2 available

Tool execution completed with the following status: Device not found

Press any key to exit.

We also trid to upgrade i40e driver with following results:

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 0 mac aa:bb:cc:aa:00:01 vlan 11

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 1 mac aa:bb:cc:aa:01:01 vlan 31

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 2 mac aa:bb:cc:aa:00:05 vlan 15

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 3 mac aa:bb:cc:aa:01:05 vlan 35

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 0 mac aa:bb:cc:bb:00:01 vlan 51

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 1 mac aa:bb:cc:bb:01:01 vlan 71

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 2 mac aa:bb:cc:bb:00:05 vlan 55

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 3 mac aa:bb:cc:bb:01:05 vlan 75

- if you retry the commands 2 or 3 times they complete succesfully

- during boot the cards are not configured.. Boot hangs with messages.

Hard reset is needed.

BUG: scheduling while atomic: ip/6585/0x00000200

BUG: scheduling while atomic: ip/6585/0x00000200

In what correlation are i40e and i40evf drivers are? Can newest version i40evf driver work with older version of i40e or vice versa?

Pls find attached dmesg log.

BR, Mate

Copy link

3,662 Views

Hi Mate,

Thank you for the additional information provided. There is this "Cisco(R) Ethernet Converged NIC 5.04 1583 00:006 Update not XL710-QDA2 available" entry. Is this an onboard OEM NIC?

Can you help clarify?

rgds,

wb

Copy link

3,662 Views

This is PCIe NIC bought with Cisco UCS server, Cisco PN is UCSC-PCIE-ID40GF (Intel XL710 dual-port 40G QSFP+ NIC).

We checked some statistics with ethtool (report without zeros bellow), no drops or errors, but port.VF_admin_queue_requests has some big value. Is this maybe indicates the problem?

KVM-1:~# ethtool -S enp6s0f0 | egrep -v :\ 0 | column

NIC statistics: port.rx_multicast: 212105 port.rx_size_255: 8 port.VF_admin_queue_requests: 206741

port.rx_bytes: 51293682 port.tx_multicast: 3529 port.rx_size_511: 5173 port.fdir_atr_status: 1

port.tx_bytes: 44178577 port.rx_broadcast: 100 port.tx_size_64: 139229 port.fdir_sb_status: 1

port.rx_unicast: 445716 port.tx_broadcast: 59 port.tx_size_127: 447849

port.tx_unicast: 583502 port.rx_size_127: 652740 port.tx_size_255: 12

KVM-1:~#

Copy link

3,662 Views

Hi Mate,

Thank you for the clarification. We will try check and have you also contacted Cisco about this as this is an OEM NIC.

rgds,

wb

Copy link

3,662 Views

Cisco says that Ubuntu is not supported... :-(

Is this OEM NIC so much different and modified that regular Intel driver will not work correctly?

Copy link

3,662 Views

Hi Mate,

As this is OEM NIC we recommend to use the supported OS and customized driver from the OEM vendor. Intel driver are generic version which we do not guarantee 100% compatibility. For the vf and pf driver, it is advisable to load the latest version as it will include fixes for some issues. Please do let me know if you have other inquiries.

Rgds,

wb

Copy link

3,662 Views

Hi Mate,

Please feel free to update me if you have other inquiries.

rgds,

wb

Copy link

3,662 Views

Hi,

You said that NIC is OEM and that we should we should use customzed driver from vendor. it is interesting that Cisco says that NIC is Intel and we should use standard Intel drivers.

BR, Mate

Copy link

3,662 Views

Hi Mate,

AS mentioned in your previous post, the NIC is bought with Cisco UCS server, Cisco PN is UCSC-PCIE-ID40GF (this is a Cisco part number). We will help check if there is any information. Thanks.

rgds,

wb

Copy link

3,662 Views

Hi Mate,

Can you try again the most current driver and firmware?

Firmware from NVMUpdatePackage (5.05):

Linux: https://downloadcenter.intel.com/download/25791

All: https://downloadcenter.intel.com/download/24769

Latest Driver 2.0.23

Link: https://sourceforge.net/projects/e1000/files/i40e%20stable/

Please provide the following information:

1) run the ethtool -I output and share the information.

2) Please provide the network adapter Serial number and PBA number (these can be found at the white sticker on the physical network adapter, the format goes like

15 alphanumeric + 6 alphanumeric + 6-3 digits)

Thanks,

wb

Copy link

3,662 Views

Hi Mate,

Please feel free to provide the information.

Thanks,

wb

Copy link

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.