Issue with HP DL380 Gen10 server with Intel XXV710-2

[KPoku]

We are using HP DL380 Gen10 servers each with two Intel XXV710-2 NIC's in our data center with SR-IOV feature.

OS on servers is Ubuntu:

VERSION="16.04.6 LTS (Xenial Xerus)"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 16.04.6 LTS"

VERSION_ID="16.04"

Linux ri-cgn-kvm1 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Intel i40e drivers are up date:

 i40e: Intel(R) 40-10 Gigabit Ethernet Connection Network Driver - version 2.8.43

Firmware version on servers:

iLO 5   1.40 Feb 05 2019   

System ROM   U30 v2.04 (04/18/2019)  

Intelligent Platform Abstraction Data   8.9.0 Build 38  

System Programmable Logic Device   0x2E   

Power Management Controller Firmware   1.0.4  

Power Supply Firmware   1.00   Bay 1   

Power Supply Firmware   1.00   Bay 2   

Innovation Engine (IE) Firmware   0.2.0.11   

Server Platform Services (SPS) Firmware   4.1.4.251   

Redundant System ROM   U30 v2.00 (02/02/2019)  

Intelligent Provisioning   3.30.213   System Board   

Power Management Controller FW Bootloader   1.1  

HPE Smart Storage Battery 1 Firmware   0.70   Embedded Device   

HPE Ethernet 1Gb 4-port 331i Adapter - NIC   20.14.54   

HPE Smart Array P408i-a SR Gen10   1.98   Embedded RAID   

Intel Ethernet Network Adapter XXV710-2   1.2154.0   PCI-E Slot1

Intel Ethernet Network Adapter XXV710-2   1.2154.0   PCI-E Slot4   

Embedded Video Controller   2.5   Embedded Device

In average once per week we get same error on different server in our data center on iLO:

1. PCI Bus   Uncorrectable PCI Express Error Detected. Slot 4 (Segment 0x0, Bus 0xAE, Device 0x0, Function 0x0). Uncorrectable Error Status: 0x100000   05/22/2019 04:53:12   1   Hardware

2. System Error   Unrecoverable I/O Error has occurred. System Firmware will log additional details in a separate IML message entry if possible.   05/22/2019 04:53:12   1   Hardware

3. CPU   Uncorrectable Machine Check Exception (Processor 2, APIC ID 0x00000040, Bank 0x00000006, Status 0xBB800000'00000E0B, Address 0x00000000'00000000, Misc 0x00000000'AE000000).

In this case unable server is not responding, even the console on iLO doesn't work and only reboot helps.

Error is related to PCI-E slot where Intel cards are connected.

Also there are issues with SR-IOV, when VM that is using VF stop to process traffic and we i see this in kern.log file:

Jul 14 06:28:47 ri-cgn-kvm4 kernel: [803323.350238] i40e 0000:af:00.0: TX driver issue detected on VF 1

Jul 14 06:28:47 ri-cgn-kvm4 kernel: [803323.350241] i40e 0000:af:00.0: Use PF Control I/F to re-enable the VF

Did anyone had this issues?

I've tried to contact HP support but as it seems we are using NIC that is offically unsupported with HP server.

Regards,

Kresimir

[Caguicla_Intel]

Hello Kresimir,

Thank you for the update.

We are sorry to hear that driver update and turning off the offloading feature wasn't of help to fix the issue.

Please allow us to further investigate on this matter. Rest assured that we will provide an update within 1-3 business days.

Hoping for your patience.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Kresimir,

Good day!

While we are still checking on this, kindly share the the ethtool -k and ethtool -c output to show the settings on the XXV710.

Looking forward to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[KPoku]

Hello Crisselle,

i've attached the text file containing commands output.

Kind regards,

Kresimir

[Caguicla_Intel]

Hello Kresimir,

We appreciate your prompt reply. We'll check on this further and provide an update within 1-3 business days.

Thank you for the patience.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Kresimir,

Thank you for the patience on this matter.

Kindly check the details below for the additional information we need.

1. You have mentioned that HP told you that this Adapter is not supported on the system. Kindly share why are you using it on the system?
2. Have you tried to use a different adapter on the system? If yes, does the same issue(s) occur?

Looking forward to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Kresimir,

Kindly provide the details that we requested for us to continue to check on your request.

Your prompt reply is highly appreciated.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[KPoku]

Hello Crisselle,

thank you for your feedback.

Regarding your questions, could you please explain question 1.?

I'm not sure how answer to this question we'll be helpful resolving this issue.

Regarding your second question, we are in process of testing Mellanox card on different servers but with same setup (same hypervisor, same virtual machines.

Kind regards,

Kresimir

[Caguicla_Intel]

Hello Kresimir,

We appreciate your reply.

Since the card is not supported/tested/compatible on the HP server, this factor may/may not affect the performance of the Network Card. Can you share why are you still using the card on the system? Is there any specific feature that only XXV710 has?

We'd like to double check if the Mellanox card is supported on the HP server? Kindly share the exact model of the card.

We have managed to check the specifications of the HP server. Based on the specifications, it has an embedded 4x1GbE. Have you tried to use this to further isolate the issue?

Awaiting to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Kresimir,

We'd like to follow up the requested information for us to further assist you on this matter.

We look forward to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[KPoku]

Hello Crisselle,

we are using Intel card because it has 25Gbps ports, so we cannot use embedded 4x1Gbps, but this is not the point.

There is no need for you to check if Mellanox is supported, as we already know that is supported.

Can you suggest what else we can do to try to resolve this issue?

If you do not have any more idea, you can go ahead and close this case.

Kind regards,

Kresimir

[Caguicla_Intel]

Hello Kresimir,

Thank you for the response.

Rest assured that we are still checking on this for any other recommendations we can provide. We will give you an update you within 1-3 business days. Thank you for your time on this matter

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Kresimir,

Please be informed that we are still checking your query. We will get back to you to provide an update within 1-3 business days.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Kresimir,

We apologize for the delay on this matter.

Kindly provide the PBA and serial number of the adapters. You may refer to the link below on where to find the PBA number. You may also provide photos of the adapters focusing on the markings (white sticker) found on the physical card for us to double check on it.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html

We'd also like to confirm if the latest BIOS is loaded on your system.

Looking forward to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[KPoku]

Hi Crisselle,

thank you for your assistance, but i must inform you that there is no need to troubleshoot this matter any further.

We decided to use Mellanox ConnectX-4 Lx NIC instead.

Kind regards,

Kresimir

[Caguicla_Intel]

Hello Kresimir,

You are welcome.

We hope that the Mellanox card won't give your system any issues. Since there is no need to troubleshoot this request any further, please be informed that we will now proceed with closure. Should you have any other concern or assistance needed in the future, please do not hesitate to post a new question.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel