X710 Auto-negotiation / advertised speed issue firmware issue?

[Echo-Bravo]

Hello,

I've run into an auto-negotiation endless loop between two directly connected Intel Converged Network Adapter X710 devices connected over fiber. The following observations can be seen.

• This problem is observed after system start-up or not at all. 
• This problem does not occur often, approximately 5% of the time. 
• When this problem happens, the link is observed to constantly go up and down.  Seen similarly on both ends.  
• This problem seems to happen when both X710 are using firmware version 6.01, but we've been unable to replicate when using 6.01 (X710 A) and 5.05 (X710 B). While this seems very anecdotal, we've tried multiple systems with the X710 cards running 5.05 and multiple systems with the 6.01.  
• We can work around the issue by forcing the advertised speed to only one option (10GBASE-SR)

Again when we connect a X710 running 5.05 to a X710 running 6.01 we are unable to replicate this problem.  When we connect a X710 running 6.01 to a X710 running 6.01 we see this problem approximately 5% of the time. We have tried different hardware sets with this. We have tried different SFPs. The behavior seems consistent, although surprising.

Would the firmware version have anything to do with this problem? It seems to be surprising the matching firmware would have a different/more negative performance than mismatched. But this is the only variable.

Is it recommended to upgrade the firmware on the X710 devices?  Regularly?

Understand that firmware changes are considered propriety, is there any knowledge if those updates would address this issue? 

System Details: All Linux based servers. All identical hardware sets.  Supermicro servers X10-XDRi-t motherboards. 
Some systems run an Intel X710 with firmware version 5.05
Some systems run an Intel X710 with firmware version 6.01
All systems are using the Linux i40e driver version 2.9.21
All X710 are using Intel E10GSFPSRG1P5 SFPs.  

Thanks.

[RyanFeeko]

Hey @Echo-Bravo , just wondering if can consider to replicate the issue with latest firmware?

I found there is the latest firmware available which is 9.5. (if not mistaken) and you managed to try with 5.05 and 6.01.

Here the link if i not wrong: https://www.intel.com/content/www/us/en/download/18190/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-700-series.html

See if that can resolve your issue since latest firmware fixed many bugs.

[Echo-Bravo]

Hello,

We plan to try updating the firmware to newest and retrying with a limited set of boards/systems. 

However, I am curious on what changes were also made and how to go about getting that information. Are there any security vulnerabilities as part of the changes? 

Additionally, I wasn't sure what the stance on firmware upgrades for the NICs was.  How risky of an operation is upgrading the firmware? For example, many motherboard vendors typically recommend not changing BIOS firmware unless you have to.  Does Intel recommend keeping NIC firmware up to date or do they normally only recommend updating in specific cases?   

Thanks

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

Hope you're doing well. Thank you for clarifying the issue further. We would like to know if this product was shipped with the system? and if any hardware or software changes were made. Have you checked both NIC cards with version 6.01? Additionally, please let us know where you updated version 5.05 from. This information will help us provide more assistance.

Best Regards,

Azeem_Intel

[Echo-Bravo]

Hello,

Shipped with the system is sort of difficult to provide a simple answer.

A third-party vendor procured the Supermicro servers and all additional PCIe cards. The third-party vendor installed all the equipment and then performed some "packaging" modifications to the server assembly. We received the products directly from them.

There were no modifications to the compute hardware or software. The third-party vendor did not modify/upgrade the X710 firmware after procuring it. To date we have not modified the firmware on any of the X710 NICs as well. So, the 5.05 and 6.01 firmware are the firmware versions that were on the NICs when procured. 

Note: Some of this hardware was procured with a long lead time production parts. This means that some parts are order much sooner than they are needed. I don't have an exact date when the X710 NICs were procured by the third-party vendor.  I estimate their procurement may have happened up to 5 years ago.

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

I appreciate you for sharing all the proper details. Thank you for that. Usually, this issue can occur due to a driver or it can be setting. I recommend updating the driver once using the link provided below. Additionally, I would like to inform you that Supermicro servers come with assembled systems, so I would recommend you connect with Supermicro (OEM).

Search - Intel.com

Intel® Ethernet Adapter Complete Driver Pack

Best Regards,

Azeem_Intel

[Echo-Bravo]

We will attempt to update the driver to assess any behavioral changes. 

Are you recommending that we do not upgrade the X710 firmware at this time.  Or are you recommending we first attempt the driver update and then assess firmware updates? 

As for:

Additionally, I would like to inform you that Supermicro servers come with assembled systems, so I would recommend you connect with Supermicro (OEM).

The Supermicro (OEM) systems were not procured with the Intel X710 NICs. They were installed after OEM delivery by a third-party vendor who purchased the Intel Retail X710.  Board #J11367-XXX. 

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

Thank you for your response. Could you please share a snapshot of your Ethernet card with the serial number visible on both sides? We will check with our team to see what further support we can provide. I will try my best to resolve your query. Thank you.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

This is the first follow-up regarding the issue (X710 Auto-negotiation / advertised speed issue firmware issue?) you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided

Best Regards,

Azeem_Intel

[Echo-Bravo]

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

Thank you so much for sharing these snapshots. Although they are not very clear, we recommend updating to the latest driver from the Intel site if the NIC is experiencing issues with the current driver. Please check if updating the driver resolves the issue or if the problem persists.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

This is the first follow-up regarding the issue (X710 Auto-negotiation / advertised speed issue firmware issue?) you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

This is the second follow-up regarding the reported issue. We're eager to ensure a swift resolution and would appreciate any updates or additional information you can provide.

Best Regards,

Azeem_Intel

[Echo-Bravo]

Azeem,

We are working to get the most up to date driver included as part of our kernel.  We are running a RHEL7 based system where the kernel has been recompiled to build everything into the kernel (no modules).  This has added some extra effort to building the driver into the kernel.  

Understand the driver is the request first, but as asked before, what is Intel's recommendation on upgrading the firmware on the cards?

Thanks

[Simon-Intel]

Hi Echo-Bravo,

Thank you for your update on the driver. Regarding the firmware, we would like to let you know that it depends on the firmware version the OS will support.

Please find the link to the firmware: Non-Volatile Memory (NVM) Update Utility for Intel Ethernet Network Adapter 700 Series.

We look forward to hearing from you.

Regards,

Simon

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

This is the first follow-up regarding the issue (X710 Auto-negotiation / advertised speed issue firmware issue?) you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

This is the second follow-up regarding the reported issue. We're eager to ensure a swift resolution and would appreciate any updates or additional information you can provide.

Best Regards,

Azeem_Intel

[Echo-Bravo]

We updated our drive to use the newest driver. 

We are able to replicate the same problem.  

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

Thank you for your response. Could you please share updated version snapshots and, if available, any error snapshots? We will check with our internal team and get back to you with a resolution.

Best Regards,

Azeem_Intel

[Echo-Bravo]

I can provide updated version screenshots.  Is a scanned screenshot of Ethtool output sufficient?  

The other error snapshot is hard, what we tend to see from Ethtool is it swapping between showing a connection and not showing a connection.  I'm not sure we get any Kerenel dmesg output that is of any value.  I'll have to check to see if we can capture that.  

[Azeem_Intel]

Hi Echo-Bravo,

Greetings for the day!

This is the first follow-up regarding the issue (X710 Auto-negotiation / advertised speed issue firmware issue?) you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided

Best Regards,

Azeem_Intel