Ubuntu 16.04 and Intel XL710 SR-IOV - Packet drops

[MGrba]

We have server running Ubuntu 16.4 KVM with Intel XL710 40Gbps NIC with SR-IOV on top of it.

On top of each 40G interfaces are created 4 SR-IOV VFs which are associated to VMs.

Problem is that after traffic load increases (approx 6Gbps on physical intf) we are experencing increased RTT and traffic drops.

VMs looks OK, switches and rest of the network also looks good and we are suspecting to SR-IOV.

How can I verify potential traffic drops on SR-IOV?

Has anybody similar expirience?

Kernel: 4.4.0-66-generic

driver=i40e driverversion=1.4.25-k duplex=full firmware=5.04

BR, Mate

[idata]

Hi Mate,

Please try upgrading the NVM firmware to version 5.0.5 https://downloadcenter.intel.com/download/25791/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Adapters-Linux-?product=83967 Understand you are using Ubuntu OS, the driver we have is for Linux*, version 1.6.4.2 https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-Intel-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=83967 Virtual function driver version 1.6.41 https://downloadcenter.intel.com/download/24693/Intel-Network-Adapter-Virtual-Function-Driver-for-Intel-40-Gigabit-Ethernet-Network-Connections?product=83967 Please try check if the driver is applicable.Thanks,wb

[MGrba]

Hi,

tnx. I think we already tried this, but we had some problems with loading drivers. We'll try again and post results here.

Meanwhile, how can we check is there any packet drops/errors on VF and PF?

ethtool -S enp6s0f0 shows no errors or drops

BR, Mate

[idata]

Hi Mate,

Thank you for the reply, please feel free to update me the result. I will check on this.rgds,wb

[MGrba]

Hi,

in first post I forgot to write down full NIC model, it is Intel XL710-QDA2.

We tried to update FW and got following error/status (Update not available):

:

./nvmupdate64e

Intel(R) Ethernet NVM Update Tool

NVMUpdate version 1.28.19.4

Copyright (C) 2013 - 2016 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

Inventory in progress. Please wait [|.........]

Num Description Ver. DevId S:B Status

=== ======================================== ===== ===== ====== ===============

01) Intel(R) I350 Gigabit Network Connection 1.99 1521 00:001 Update not

available

02) Cisco(R) Ethernet Converged NIC 5.04 1583 00:006 Update not

XL710-QDA2 available

Tool execution completed with the following status: Device not found

Press any key to exit.

We also trid to upgrade i40e driver with following results:

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 0 mac aa:bb:cc:aa:00:01 vlan 11

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 1 mac aa:bb:cc:aa:01:01 vlan 31

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 2 mac aa:bb:cc:aa:00:05 vlan 15

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 3 mac aa:bb:cc:aa:01:05 vlan 35

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 0 mac aa:bb:cc:bb:00:01 vlan 51

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 1 mac aa:bb:cc:bb:01:01 vlan 71

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 2 mac aa:bb:cc:bb:00:05 vlan 55

Segmentation fault (core dumped)

root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 3 mac aa:bb:cc:bb:01:05 vlan 75

- if you retry the commands 2 or 3 times they complete succesfully

- during boot the cards are not configured.. Boot hangs with messages.

Hard reset is needed.

BUG: scheduling while atomic: ip/6585/0x00000200

BUG: scheduling while atomic: ip/6585/0x00000200

In what correlation are i40e and i40evf drivers are? Can newest version i40evf driver work with older version of i40e or vice versa?

Pls find attached dmesg log.

BR, Mate

[idata]

Hi Mate,

Thank you for the additional information provided. There is this "Cisco(R) Ethernet Converged NIC 5.04 1583 00:006 Update not XL710-QDA2 available" entry. Is this an onboard OEM NIC?Can you help clarify? rgds,wb

[MGrba]

This is PCIe NIC bought with Cisco UCS server, Cisco PN is UCSC-PCIE-ID40GF (Intel XL710 dual-port 40G QSFP+ NIC).

We checked some statistics with ethtool (report without zeros bellow), no drops or errors, but port.VF_admin_queue_requests has some big value. Is this maybe indicates the problem?

KVM-1:~# ethtool -S enp6s0f0 | egrep -v :\ 0 | column

NIC statistics: port.rx_multicast: 212105 port.rx_size_255: 8 port.VF_admin_queue_requests: 206741

port.rx_bytes: 51293682 port.tx_multicast: 3529 port.rx_size_511: 5173 port.fdir_atr_status: 1

port.tx_bytes: 44178577 port.rx_broadcast: 100 port.tx_size_64: 139229 port.fdir_sb_status: 1

port.rx_unicast: 445716 port.tx_broadcast: 59 port.tx_size_127: 447849

port.tx_unicast: 583502 port.rx_size_127: 652740 port.tx_size_255: 12

KVM-1:~#

[idata]

Hi Mate,

Thank you for the clarification. We will try check and have you also contacted Cisco about this as this is an OEM NIC.rgds,wb

[MGrba]

Cisco says that Ubuntu is not supported... :-(

Is this OEM NIC so much different and modified that regular Intel driver will not work correctly?

[idata]

Hi Mate,

As this is OEM NIC we recommend to use the supported OS and customized driver from the OEM vendor. Intel driver are generic version which we do not guarantee 100% compatibility. For the vf and pf driver, it is advisable to load the latest version as it will include fixes for some issues. Please do let me know if you have other inquiries.Rgds,wb

[idata]

Hi Mate,

Please feel free to update me if you have other inquiries.rgds,wb

[MGrba]

Hi,

You said that NIC is OEM and that we should we should use customzed driver from vendor. it is interesting that Cisco says that NIC is Intel and we should use standard Intel drivers.

BR, Mate

[idata]

Hi Mate,

AS mentioned in your previous post, the NIC is bought with Cisco UCS server, Cisco PN is UCSC-PCIE-ID40GF (this is a Cisco part number). We will help check if there is any information. Thanks. rgds,wb

[idata]

Hi Mate,

Can you try again the most current driver and firmware? Firmware from NVMUpdatePackage (5.05): Linux: https://downloadcenter.intel.com/download/25791All: https://downloadcenter.intel.com/download/24769 Latest Driver 2.0.23Link: https://sourceforge.net/projects/e1000/files/i40e%20stable/ Please provide the following information:1) run the ethtool -I output and share the information.2) Please provide the network adapter Serial number and PBA number (these can be found at the white sticker on the physical network adapter, the format goes like15 alphanumeric + 6 alphanumeric + 6-3 digits)Thanks,wb

[idata]

Hi Mate,

Please feel free to provide the information.Thanks,wb