X710 link flapping

[hlumca]

I have a X710 connected to a Cisco switch using a Cisco DAC cable. The switch is reporting linkFlapErrDisabled. How can I resolve this? I have upgraded the NVM to 8.20 running under CentOS 7.9

[Caguicla_Intel]

Hello hlumca,

Thank you for posting in Intel Ethernet Communities. 

1. Please provide front and back photos of the X710 adapters focusing on the markings (white stickers). This will allow us to identify if you are using a retail or Original Equipment Manufacturer(OEM) of Intel Ethernet Adapter. We will send an email for you to attach the photos of the adapter. Please be reminded not to share the photos here to avoid exposing confidential information.

2. Have you tried testing or using Compatible SFP+ Modules, SFP Modules, and Cables for Intel® Ethernet Server Adapter X710 Series as listed on the link below? 

Mentioned on the article below that Other brands of SFP+ optical modules don't work with the Intel® Ethernet Converged Network Adapter (Intel® Ethernet CNA) X710 Series. You must use the listed Intel® SFP+ SR or LR Optics.

https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

3. What is the current driver version used? 

4. Have you also tried connecting the cable on different port of the switch?

Looking forward to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[hlumca]

The pictures have been emailed.

We started off with an Intel E65689-005 / FTLX8574D3BCV-IT plugged in the the X710 and a Cisco SFP-10G-SR plugged in to the Cisco switch. When we saw the link flapping error, we tried using a Cisco DAC (SFP-H10GB-CU1M) but the flapping error was still present. We then tried different ports on the Cisco switch with the same result.

The driver version is 2.8.20-k which is associated with the CentOS 7.9 kernel package.

[Caguicla_Intel]

Hello hlumca,

Thank you for the reply.

We've sent an email to you as we are not able to extract the markings from the photos provided. Kindly check your inbox.

Additionally, please provide the following details.

1. You mentioned that driver version is 2.8.20-k which is associated with the CentOS 7.9 kernel package. Can you confirm if you already tried the latest driver from the link below? 

Intel® Network Adapter Driver for PCIe* 40 Gigabit Ethernet Network Connections under Linux*

https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-under-Linux-

2. We checked if FTLX8574D3BCV-IT is listed as compatible module for this adapter and unfortunately, this is not on the list. 

3. Please provide the exact model of the switch used. 

Looking forward to your reply.

In case we don't receive your reply, we will reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello hlumca,

Thank you for the effort in sending us a clearer photos of the markings of your Ethernet adapter.

We would also like to follow up the additional information requested below as we will further check this with our engineers.

1. You mentioned that driver version is 2.8.20-k which is associated with the CentOS 7.9 kernel package. Can you confirm if you already tried the latest driver from the link below? 

Intel® Network Adapter Driver for PCIe* 40 Gigabit Ethernet Network Connections under Linux*

https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-under-Linux-

2. Please provide the exact model of the switch used. 

Looking forward to hear from you. 

We will follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel® Customer Support

[hlumca]

We have not tried a different driver version. The same driver version works fine on a different server.

The switch is a Cisco 5672UP.

Regarding the FTLX8574D3BCV-IT, it is in the list of qualified SFPs for the X710.

As mentioned above, we have a different server that works fine with the X710 card. One difference is that the X710 card has firmware 6.01. Can you provide a downgrade file so that we can try out 6.01?

[Caguicla_Intel]

Hello hlumca,

Thank you for the swift response.

We sincerely apologize as we mentioned that FTLX8574D3BCV-IT is not listed as compatible SFP for X710. We have double checked and confirmed that this is on the list based on link below.

https://compatibleproducts.intel.com/ProductDetails?activeModule=Intel%C2%AE%20Ethernet

Since you mentioned that you have a different server that works fine with the X710 card and the only difference is the Firmware version, can you please confirm if the X710 is connected on the same switch which was Cisco 5672UP and uses the same model of optical modules? 

Awaiting to hear from you soon. 

We will reach out after after 3 business days in case we don't receive an update. 

Best regards,

Crisselle C.

Intel® Customer Support

[hlumca]

Yes, the other server is connected to the same switch and using the same optical modules.

[Caguicla_Intel]

Hello hlumca,

Thank you for the confirmation.

While we are checking this with our engineers, we hope you won't mind extracting System Support Utility logs from 2 servers (one with and without link flapping issue). This will allow us to check your Adapter details and configuration that would help in further investigating the issue. Kindly refer to the Help Guide below on how to extract the Intel® System Support Utility logs. We will send an email for you to attach the logs and please be reminded not to share the logs on this thread to avoid exposing confidential information.

https://www.intel.com/content/www/us/en/support/articles/000008563/software/software-applications.html

Looking forward to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello hlumca,

Thank you for providing the requested information via email. 

Please allow us to forward this to our engineers for further investigation. We will get back to you as soon as we heard an update from them but no later thank 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello hlumca,

Apologies for the delay on our update.

Please be informed that this request is still under investigation. Additionally, our engineering team would like to clarify below information for further investigation

1.) When did you first encountered the linkFlapErrDisabled error? Is it only after updating the NVM to 8.20 or since first integration? 

2.) Is there any hardware or software changes before the issue occur?

Looking forward to your reply.

In case we don't receive your reply, we will reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[hlumca]

The X710 was installed recently in a new system. The linkFlapE condition happened immediately. The X710 had NVM 7.00 at the time. I installed NVM 8.20 but the linkFlapE condition was still happening.

[Caguicla_Intel]

Hello hlumca,

Thank you for the swift response.

Please allow us to forward your reply to our higher level. We will get back to you as soon as we heard an update from them but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello hlumca,

Apologies for the delay on this matter.

Our engineering team is requesting for the dmesg.log while the flapping is occurring. We hope you don't mind sharing this with us for further investigation.

Hoping to hear from you soon.

We will follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel® Customer Support

[hlumca]

On the Cisco switch, I did a shutdown / no shutdown. On the CentOS 7.9 server, dmesg had

[1127156.095900] i40e 0000:0e:00.0 enp14s0f0: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[1127159.733573] IPv6: ADDRCONF(NETDEV_CHANGE): enp14s0f0: link becomes ready
[1127161.703070] i40e 0000:0e:00.1 enp14s0f1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[1127165.340740] IPv6: ADDRCONF(NETDEV_CHANGE): enp14s0f1: link becomes ready

On the Cisco switch, show interface status reported the ports toggling between connected and notconnec for a few minutes. Eventually the ports reverted to the linkFlapE state. On the CentOS 7.9 server there were no additional messages after the four lines above.

[Caguicla_Intel]

Hello hlumca,

Thank you for the prompt reply.

Please allow us to forward this to our engineers. We will get back to you as soon as we heard an update from them but no later than 2-4 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello hlumca,

I hope you're doing great!

Based on our engineering, the log provided is only a snapshot. Will you be able to provide the whole or complete dmesg.log for us to continue with the investigation?

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

[hlumca]

I was going to provide you with a dmesg file from a freshly booted server. However, the link flapping issue went away when I did the following.

1. Power off the server containing the X710
2. On the Cisco switch, I did a "shutdown" of the ports
3. On the Cisco switch, I did a "no shutdown" of the ports
4. Power on the server containing the X710

After starting CentOS 7.9, the linkFlapE condition is no longer being reported by the Cisco switch. I guess you can go ahead with closing this case. Thanks for your help.

[Caguicla_Intel]

Hello hlumca,

Thank you for the update. 

We are glad to hear that issue has been resolved. We will now proceed closing this request. Thank you for your time and cooperation throughout the process. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

May you have an amazing weekend ahead!

Best regards,

Crisselle C.

Intel® Customer Support