Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

E810-XXVDA2 fails to link up due to invalid FEC settings

MisterJ
Beginner
8,452 Views

Hi,

 

System details

I currently have 3 identical setups that all exhibit the same problem. Here is the configuration

  • OS: ESXi 8.0.0 build 20513097 (latest 8.0a release)
  • CPU: Intel Xeon E5-2697 v4 
  • Nic: E810-XXVDA2 in a PCIe Gen3 x8 slot
  • Cables: HPE 845418-B21 QSFP28 breakout to 4x SFP28
  • PBA: K57775-013
  • Switch: Celestica DX010 32x100G running SONiC NOS
  • ESXi Driver: 1.10.5.0-1OEM.702.0.0.17630552
  • UEFI Driver: 4.0.25
  • NVM: 4.10
  • NVM ID: 0x80015188

 

Problem Statement

I have discovered a few issues with getting this nic to work on ESXi 8.

 

The biggest issue is the nic does not link up to my switch at 25G speeds. But is capable of linking up at 10G with a different breakout cable. I discovered that the "intnetcli" tool fails when trying to disable "Auto-FEC" mode (which I think is the culprit of the link issue):

 

[root@hyper-core7:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1       Auto-FEC  No-FEC

 vmnic2       Auto-FEC  No-FEC

[root@hyper-core7:~] esxcli intnet fec set -m "No-FEC" -n vmnic2

Requested FEC mode is not supported. Refer to dmesg for more details.
[root@hyper-core7:~] dmesg | grep icen | tail -4
2023-01-13T18:53:46.313Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.0: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:53:46.314Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.1: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:55:03.942Z cpu13:3489616)WARNING: icen: icen_SetFec:13494: 0000:04:00.1: Requested FEC setting is not supported with the current link topology, Status: VMK_NOT_SUPPORTED
2023-01-13T18:55:03.942Z cpu13:3489616)icen: icen_SetFec:13502: 0000:04:00.1: Supported Fec modes: Auto-FEC
[root@hyper-core7:~] 

 

 

Workaround

  1. Boot into Windows OS
  2. Confirm links are still down
  3. Use Advanced Networking Properties to disable "Auto-FEC" mode on both ports
  4. Disable and Enable both links, wait for them to come up
  5. Warm reboot into ESXi

When I do this I no longer see "Auto-FEC" mode enabled in "intnetcli" tool on the ESXi hosts and the links stay up:

 

[root@hyper-core5:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1         No-FEC  No-FEC

 vmnic2         No-FEC  No-FEC

 

However, this only works until the system is cold booted. So while this does get the systems up and running it is not ideal and leads me to not want to scale this solution out to more systems (as this would be very tedious on 3 systems let alone a rack full of systems). 

 

Other issues:

  1. The latest version of "intnetcli" does not support ESXi 8 (only 6.7 up to 7.0, doesn't even support 7.1). Intel® ESXCLI Plug-In for Managing Intel® Ethernet Network Adapters
    1. I was able to extract the VIB and force install it to bypass the warnings

 

Open Questions:

  1. Is there a more permanent way to override the "Auto-FEC" mode (with an override in the EEPROM/NVM maybe)?
  2. Is there a way to force set the FEC mode (bypassing the validation)?

 

Additional Comments:

  1. No SFPs are detected on E810-xxvda4t - Intel Communities: This question was never resolved but the symptoms look very similar. 
  2. HPE put out an advisory about this adapter having this exact same issue. However, I am unable to change the adapter FEC setting within ESXi and the option is missing from preboot/HII. Document - Advisory: HPE Network Adapters - Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter for HPE May Not Link With Switches at 25Gb | HPE Support 

 

0 Kudos
23 Replies
MisterJ
Beginner
7,760 Views

Actually, while trying the workaround on the other 2 systems I noticed that I could only get both ports linked up when "Auto configuration of No FEC" is set to "Enabled {1}" and bringing the nics up in physical phy order. I am not sure why I needed to do this but I was only able to get the higher numbered port up otherwise. This might be a red herring but thought I would share it just incase it does make a difference to anyone else who finds this post.

 

Screenshot 2023-01-13 at 1.41.43 PM.png

 

The outcome in ESXi is still the same as it only shows "No-FEC" as the available options:

esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1         No-FEC  No-FEC

 vmnic2         No-FEC  No-FEC
0 Kudos
Mike_Intel
Moderator
7,716 Views

Hello MisterJ,


Thank you for posting in Intel Ethernet Communities. 


For us to further check the issue, please provide the following details.


  1. Please send photos of the network card on both sides for us to validate the card.
  2. Can you share the link of your network driver?


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support 


0 Kudos
MisterJ
Beginner
7,699 Views

Hi Michael,

 

I've attached the requested photos.

 

For the driver link, I grabbed the latest available for the card on VMware's Hardware Compatibility List. Looks like VMware's HCL is currently down for maintenance. So I will post a link to the HCL pager later today. But here is the raw download link, pulled from my browsers download history:

Regards,

Justin

0 Kudos
Mike_Intel
Moderator
7,681 Views

Hello MisterJ,


Thank you for the prompt reply.


Can you try if you can generate the SSU logs of your system for us to further check this issue?


Kindly download the SSU utility here:


https://www.intel.com/content/www/us/en/download/18895/intel-system-support-utility-for-the-linux-operating-system.html?wapkw=ssu


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support 


0 Kudos
MisterJ
Beginner
7,656 Views

Hi Michael,

 

Sure, I've attached the SSU output. However, I am not sure it will be very helpful since I had to switch to Windows to run it (since there is not an ESXi version). So the information contained in the output is not representative of the values set within ESXi. I warm rebooted into Windows so the nic should of retained it's state across the reboot. 

 

Also, I've reconfigured the system to have 1 port attached to a non-breakout cable and 1x port attached to a breakout cable. This issue is only seen when using breakout cables.

 

Regards,

Justin

0 Kudos
Mike_Intel
Moderator
7,643 Views

Hello MisterJ,


Thank you for sending the SSU that we requested. While we are checking the logs, can you try the steps in this link:


https://www.intel.com/content/www/us/en/support/articles/000087329/ethernet-products.html


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support 


0 Kudos
MisterJ
Beginner
7,498 Views

Hi Michael,

The page you linked doesn't help as it does not contain any steps to actually change the FEC mode. The only way I am able to get this to work is to use cables that are less than 3m long as this doesn't require FEC. I have a 3m cable (that states it is made for Intel nics) and it fails to establish a link due to not being able to turn on the FEC settings. But a cable from the same manufacturer that is 2m links up fine.

 

This appears to be a ESXi driver or firmware issue with auto-FEC mode as the Windows driver does not have this same issue. It is not practical to have to boot into Windows just to bring the nics up and then warm-reboot into ESXi. 

 

Also, using cables smaller than 3m is not a solution either as this is not quite long enough to reach from the top of a rack to the bottom. 

 

Regards,

Justin

0 Kudos
Faez_Intel
Moderator
7,472 Views

Hi Justin,


Good day and thanks for the clarification.

 

You mentioned on using a 2m cables seems to be working fine. However, can you clarify if you meant the 2m cables can only work  if you go into Windows first and warm boot the ESXi or does the 2m cables is instantly detected in the ESXi? 

 

Also, may I know if previously you are facing the same issue with a lower version of ESXi? (eg: ESXI 7.0, 6.5)

 

If you have any questions, please do not hesitate to let us know and we will gladly assist you. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

 

Best regards,

Faez_Intel

Intel Customer Support

 

 

 

0 Kudos
Mike_Intel
Moderator
7,589 Views

Hello MisterJ,

 

I hope you're having a wonderful day. I am just sending a soft follow up to check if you already tried the steps in the link that we provided. Please send us an update for us to further assist you if needed. We will wait for your update.

 

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

 

Best regards,

Michael L.

Intel® Customer Support 


0 Kudos
IntelSupport
Community Manager
7,520 Views

Hello MisterJ,

 

Good day and I hope this message finds you well

I am just sending a soft follow up to check if you already tried the steps in the link that we provided. Please send us an update for us to further assist you if needed. We will wait for your update.

 

If you have any questions, please do not hesitate to let us know and we will gladly assist you. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,


Faez_Intel

Intel Customer Support


0 Kudos
Faez_Intel
Moderator
7,415 Views

Hi Justin, 

 

Good day and I hope this message finds you well!

 

This is just a follow up regarding the information we requested for us to proceed checking on your query. If you have additional questions or clarifications, feel free to let us know.

 

Should there be no response from you, I’ll make sure to reach out on . Let us know if there are any preferred date/time to reach you so that we can revert accordingly.

 

Best regards,

Faez.

Intel Customer Support 


0 Kudos
Faez_Intel
Moderator
7,371 Views

Hi Justin, 

 

Good day and I hope this message finds you well!


This is a another follow up regarding on your issue. As per our previous replies, please help to provide the details/info/ for us to proceed further with the case. 


Should there be no response from you, I’ll make sure to reach out on. Let us know if there are any preferred date/time to reach you so that we can revert accordingly.


Do feel free to inquire us if you have any other questions that you may have and we will gladly assist you accordingly.


Best regards,

 

Faez.

Intel Customer Support 



0 Kudos
Faez_Intel
Moderator
7,332 Views

Hi Justin,

 

Good day. I hope you're having a wonderful day. I am just sending a follow up regarding on your issue As per our previous replies, please help to provide the details inquired for us to proceed further with the cases. We will be waiting on your reply. As for now, we will be closing your thread.

 

If you want to continue support, please reply to this community thread and we will reopen your case or create a new one so that we can continue to support you. 


Do feel free to inquire us if you have any other questions that you may have and we will gladly assist you accordingly.


Best regards,

 

Faez.

Intel Customer Support 



0 Kudos
MisterJ
Beginner
7,323 Views

Hi Faez,

 

Sorry for the delayed response, have been tied up at work on higher priority tasks. Here are the answers for the questions you asked:

 


You mentioned on using a 2m cables seems to be working fine. However, can you clarify if you meant the 2m cables can only work  if you go into Windows first and warm boot the ESXi or does the 2m cables is instantly detected in the ESXi? 


Ah sorry for not clarifying. The 2m cables are detected in ESXi without having to go into Windows. I only need to go into Windows to get my 3m and 5m (split) cables to work.


 Also, may I know if previously you are facing the same issue with a lower version of ESXi? (eg: ESXI 7.0, 6.5)

I've only tried running ESXi 8 on this hardware setup as this is a new deployment I am testing out. I don't think downgrading will help as both versions use the same Intel driver package. It might take me a week or two to get around to downgrading the OS if you feel this is a valid data point that I should capture.

 

0 Kudos
Faez_Intel
Moderator
7,280 Views

Hi Justin,


Good day and thanks for the prompt response.


No worries on that thanks for the clarification. As for the lower version of ESXi, I was just confirming if you did encountered and saw the same issue occurred if it was running on ESXi7.0 or lower. But since you have been running on ESXi8.0 this whole time, we have escalated this to our next level to further check and assist on this matter accordingly.


Please allow us some time for our next level to replicate this issue accordingly and we will get back to you if we have any updates or further inquiries from your end.


Best regards,

Faez.

Intel Customer Support 


0 Kudos
Faez_Intel
Moderator
6,639 Views

Hi Justin,


Good day and I hope this message finds you well!


Apologies for the delayed response and we appreciate your patience on this.


We have checked and simulated on this and it seems you may need to wait for the “intnetcli” to be released for ESXi 8.0 to efficiently control the FEC.


Kindly let us know if you have any other inquiries regarding on this case.


Best regards,

Faez.

Intel Customer Support




0 Kudos
MisterJ
Beginner
6,418 Views

Thanks, Faez.

 

Do you have a tentative timeline on when an update to the "intnetcli" tool will be released? Seems like this does not follow the same release cycle as the NVM and driver updates (as they have been updated multiple times since this was last updated).

0 Kudos
Faez_Intel
Moderator
6,404 Views

Hi Justin,

 

Good day.

 

As of now we do not have any information with the release date of the "intnetcli" tool for ESXi 8. We can only know once it is announced by VMware themselves.

 

Kindly let me know if you have any other inquiries.

 

Best regards,
Faez.
Intel Customer Support

0 Kudos
Faez_Intel
Moderator
6,432 Views

Hi Justin,


Good day and I hope this message finds you well!


This is just a follow up on this thread. Please kindly let us know if you have any other assistance needed from this thread and we will assist you accordingly.


Best regards,

Faez.

Intel Customer Support


0 Kudos
Reply