- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have server running Ubuntu 16.4 KVM with Intel XL710 40Gbps NIC with SR-IOV on top of it.
On top of each 40G interfaces are created 4 SR-IOV VFs which are associated to VMs.
Problem is that after traffic load increases (approx 6Gbps on physical intf) we are experencing increased RTT and traffic drops.
VMs looks OK, switches and rest of the network also looks good and we are suspecting to SR-IOV.
How can I verify potential traffic drops on SR-IOV?
Has anybody similar expirience?
Kernel: 4.4.0-66-generic
driver=i40e driverversion=1.4.25-k duplex=full firmware=5.04
BR, Mate
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Please try upgrading the NVM firmware to version 5.0.5
https://downloadcenter.intel.com/download/25791/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Adapters-Linux-?product=83967
Understand you are using Ubuntu OS, the driver we have is for Linux*, version 1.6.4.2
https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-Intel-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=83967
Virtual function driver version 1.6.41
https://downloadcenter.intel.com/download/24693/Intel-Network-Adapter-Virtual-Function-Driver-for-Intel-40-Gigabit-Ethernet-Network-Connections?product=83967
Please try check if the driver is applicable.
Thanks,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
tnx. I think we already tried this, but we had some problems with loading drivers. We'll try again and post results here.
Meanwhile, how can we check is there any packet drops/errors on VF and PF?
ethtool -S enp6s0f0 shows no errors or drops
BR, Mate
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Thank you for the reply, please feel free to update me the result.
I will check on this.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
in first post I forgot to write down full NIC model, it is Intel XL710-QDA2.
We tried to update FW and got following error/status (Update not available):
:
./nvmupdate64e
Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.28.19.4
Copyright (C) 2013 - 2016 Intel Corporation.
WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [|.........]
Num Description Ver. DevId S:B Status
=== ======================================== ===== ===== ====== ===============
01) Intel(R) I350 Gigabit Network Connection 1.99 1521 00:001 Update not
available
02) Cisco(R) Ethernet Converged NIC 5.04 1583 00:006 Update not
XL710-QDA2 available
Tool execution completed with the following status: Device not found
Press any key to exit.
We also trid to upgrade i40e driver with following results:
root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 0 mac aa:bb:cc:aa:00:01 vlan 11
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 1 mac aa:bb:cc:aa:01:01 vlan 31
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 2 mac aa:bb:cc:aa:00:05 vlan 15
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f0 vf 3 mac aa:bb:cc:aa:01:05 vlan 35
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 0 mac aa:bb:cc:bb:00:01 vlan 51
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 1 mac aa:bb:cc:bb:01:01 vlan 71
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 2 mac aa:bb:cc:bb:00:05 vlan 55
Segmentation fault (core dumped)
root@RI-CGN-KVM-1:~# ip link set enp6s0f1 vf 3 mac aa:bb:cc:bb:01:05 vlan 75
- if you retry the commands 2 or 3 times they complete succesfully
- during boot the cards are not configured.. Boot hangs with messages.
Hard reset is needed.
BUG: scheduling while atomic: ip/6585/0x00000200
BUG: scheduling while atomic: ip/6585/0x00000200
In what correlation are i40e and i40evf drivers are? Can newest version i40evf driver work with older version of i40e or vice versa?
Pls find attached dmesg log.
BR, Mate
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Thank you for the additional information provided. There is this "Cisco(R) Ethernet Converged NIC 5.04 1583 00:006 Update not XL710-QDA2 available" entry. Is this an onboard OEM NIC?
Can you help clarify?
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is PCIe NIC bought with Cisco UCS server, Cisco PN is UCSC-PCIE-ID40GF (Intel XL710 dual-port 40G QSFP+ NIC).
We checked some statistics with ethtool (report without zeros bellow), no drops or errors, but port.VF_admin_queue_requests has some big value. Is this maybe indicates the problem?
KVM-1:~# ethtool -S enp6s0f0 | egrep -v :\ 0 | column
NIC statistics: port.rx_multicast: 212105 port.rx_size_255: 8 port.VF_admin_queue_requests: 206741
port.rx_bytes: 51293682 port.tx_multicast: 3529 port.rx_size_511: 5173 port.fdir_atr_status: 1
port.tx_bytes: 44178577 port.rx_broadcast: 100 port.tx_size_64: 139229 port.fdir_sb_status: 1
port.rx_unicast: 445716 port.tx_broadcast: 59 port.tx_size_127: 447849
port.tx_unicast: 583502 port.rx_size_127: 652740 port.tx_size_255: 12
KVM-1:~#
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Thank you for the clarification. We will try check and have you also contacted Cisco about this as this is an OEM NIC.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Cisco says that Ubuntu is not supported... :-(
Is this OEM NIC so much different and modified that regular Intel driver will not work correctly?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
As this is OEM NIC we recommend to use the supported OS and customized driver from the OEM vendor. Intel driver are generic version which we do not guarantee 100% compatibility. For the vf and pf driver, it is advisable to load the latest version as it will include fixes for some issues. Please do let me know if you have other inquiries.
Rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Please feel free to update me if you have other inquiries.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
You said that NIC is OEM and that we should we should use customzed driver from vendor. it is interesting that Cisco says that NIC is Intel and we should use standard Intel drivers.
BR, Mate
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
AS mentioned in your previous post, the NIC is bought with Cisco UCS server, Cisco PN is UCSC-PCIE-ID40GF (this is a Cisco part number). We will help check if there is any information. Thanks.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Can you try again the most current driver and firmware?
Firmware from NVMUpdatePackage (5.05):
Linux: https://downloadcenter.intel.com/download/25791
All: https://downloadcenter.intel.com/download/24769
Latest Driver 2.0.23
Link: https://sourceforge.net/projects/e1000/files/i40e%20stable/
Please provide the following information:
1) run the ethtool -I output and share the information.
2) Please provide the network adapter Serial number and PBA number (these can be found at the white sticker on the physical network adapter, the format goes like
15 alphanumeric + 6 alphanumeric + 6-3 digits)
Thanks,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Mate,
Please feel free to provide the information.
Thanks,
wb
![](/skins/images/7FC17B7B85029576C25F1E43CE255B51/responsive_peak/images/icon_anonymous_message.png)
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page