Ethernet Products
Intel® Ethernet products and technologies
Announcements
The Intel sign-in experience is changing in February to support enhanced security controls. If you sign in, click here for more information.
4173 Discussions

E810-XXVDA2 fails to link up due to invalid FEC settings

MisterJ
Beginner
411 Views

Hi,

 

System details

I currently have 3 identical setups that all exhibit the same problem. Here is the configuration

  • OS: ESXi 8.0.0 build 20513097 (latest 8.0a release)
  • CPU: Intel Xeon E5-2697 v4 
  • Nic: E810-XXVDA2 in a PCIe Gen3 x8 slot
  • Cables: HPE 845418-B21 QSFP28 breakout to 4x SFP28
  • PBA: K57775-013
  • Switch: Celestica DX010 32x100G running SONiC NOS
  • ESXi Driver: 1.10.5.0-1OEM.702.0.0.17630552
  • UEFI Driver: 4.0.25
  • NVM: 4.10
  • NVM ID: 0x80015188

 

Problem Statement

I have discovered a few issues with getting this nic to work on ESXi 8.

 

The biggest issue is the nic does not link up to my switch at 25G speeds. But is capable of linking up at 10G with a different breakout cable. I discovered that the "intnetcli" tool fails when trying to disable "Auto-FEC" mode (which I think is the culprit of the link issue):

 

[root@hyper-core7:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1       Auto-FEC  No-FEC

 vmnic2       Auto-FEC  No-FEC

[root@hyper-core7:~] esxcli intnet fec set -m "No-FEC" -n vmnic2

Requested FEC mode is not supported. Refer to dmesg for more details.
[root@hyper-core7:~] dmesg | grep icen | tail -4
2023-01-13T18:53:46.313Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.0: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:53:46.314Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.1: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:55:03.942Z cpu13:3489616)WARNING: icen: icen_SetFec:13494: 0000:04:00.1: Requested FEC setting is not supported with the current link topology, Status: VMK_NOT_SUPPORTED
2023-01-13T18:55:03.942Z cpu13:3489616)icen: icen_SetFec:13502: 0000:04:00.1: Supported Fec modes: Auto-FEC
[root@hyper-core7:~] 

 

 

Workaround

  1. Boot into Windows OS
  2. Confirm links are still down
  3. Use Advanced Networking Properties to disable "Auto-FEC" mode on both ports
  4. Disable and Enable both links, wait for them to come up
  5. Warm reboot into ESXi

When I do this I no longer see "Auto-FEC" mode enabled in "intnetcli" tool on the ESXi hosts and the links stay up:

 

[root@hyper-core5:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1         No-FEC  No-FEC

 vmnic2         No-FEC  No-FEC

 

However, this only works until the system is cold booted. So while this does get the systems up and running it is not ideal and leads me to not want to scale this solution out to more systems (as this would be very tedious on 3 systems let alone a rack full of systems). 

 

Other issues:

  1. The latest version of "intnetcli" does not support ESXi 8 (only 6.7 up to 7.0, doesn't even support 7.1). Intel® ESXCLI Plug-In for Managing Intel® Ethernet Network Adapters
    1. I was able to extract the VIB and force install it to bypass the warnings

 

Open Questions:

  1. Is there a more permanent way to override the "Auto-FEC" mode (with an override in the EEPROM/NVM maybe)?
  2. Is there a way to force set the FEC mode (bypassing the validation)?

 

Additional Comments:

  1. No SFPs are detected on E810-xxvda4t - Intel Communities: This question was never resolved but the symptoms look very similar. 
  2. HPE put out an advisory about this adapter having this exact same issue. However, I am unable to change the adapter FEC setting within ESXi and the option is missing from preboot/HII. Document - Advisory: HPE Network Adapters - Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter ... 

 

0 Kudos
13 Replies
MisterJ
Beginner
401 Views

Actually, while trying the workaround on the other 2 systems I noticed that I could only get both ports linked up when "Auto configuration of No FEC" is set to "Enabled {1}" and bringing the nics up in physical phy order. I am not sure why I needed to do this but I was only able to get the higher numbered port up otherwise. This might be a red herring but thought I would share it just incase it does make a difference to anyone else who finds this post.

 

Screenshot 2023-01-13 at 1.41.43 PM.png

 

The outcome in ESXi is still the same as it only shows "No-FEC" as the available options:

esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1         No-FEC  No-FEC

 vmnic2         No-FEC  No-FEC
Mike_Intel
Moderator
357 Views

Hello MisterJ,


Thank you for posting in Intel Ethernet Communities. 


For us to further check the issue, please provide the following details.


  1. Please send photos of the network card on both sides for us to validate the card.
  2. Can you share the link of your network driver?


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support 


MisterJ
Beginner
340 Views

Hi Michael,

 

I've attached the requested photos.

 

For the driver link, I grabbed the latest available for the card on VMware's Hardware Compatibility List. Looks like VMware's HCL is currently down for maintenance. So I will post a link to the HCL pager later today. But here is the raw download link, pulled from my browsers download history:

Regards,

Justin

Mike_Intel
Moderator
322 Views

Hello MisterJ,


Thank you for the prompt reply.


Can you try if you can generate the SSU logs of your system for us to further check this issue?


Kindly download the SSU utility here:


https://www.intel.com/content/www/us/en/download/18895/intel-system-support-utility-for-the-linux-op...


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support 


MisterJ
Beginner
299 Views

Hi Michael,

 

Sure, I've attached the SSU output. However, I am not sure it will be very helpful since I had to switch to Windows to run it (since there is not an ESXi version). So the information contained in the output is not representative of the values set within ESXi. I warm rebooted into Windows so the nic should of retained it's state across the reboot. 

 

Also, I've reconfigured the system to have 1 port attached to a non-breakout cable and 1x port attached to a breakout cable. This issue is only seen when using breakout cables.

 

Regards,

Justin

Mike_Intel
Moderator
286 Views

Hello MisterJ,


Thank you for sending the SSU that we requested. While we are checking the logs, can you try the steps in this link:


https://www.intel.com/content/www/us/en/support/articles/000087329/ethernet-products.html


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support 


MisterJ
Beginner
141 Views

Hi Michael,

The page you linked doesn't help as it does not contain any steps to actually change the FEC mode. The only way I am able to get this to work is to use cables that are less than 3m long as this doesn't require FEC. I have a 3m cable (that states it is made for Intel nics) and it fails to establish a link due to not being able to turn on the FEC settings. But a cable from the same manufacturer that is 2m links up fine.

 

This appears to be a ESXi driver or firmware issue with auto-FEC mode as the Windows driver does not have this same issue. It is not practical to have to boot into Windows just to bring the nics up and then warm-reboot into ESXi. 

 

Also, using cables smaller than 3m is not a solution either as this is not quite long enough to reach from the top of a rack to the bottom. 

 

Regards,

Justin

Faez_Intel
Employee
115 Views

Hi Justin,


Good day and thanks for the clarification.

 

You mentioned on using a 2m cables seems to be working fine. However, can you clarify if you meant the 2m cables can only work  if you go into Windows first and warm boot the ESXi or does the 2m cables is instantly detected in the ESXi? 

 

Also, may I know if previously you are facing the same issue with a lower version of ESXi? (eg: ESXI 7.0, 6.5)

 

If you have any questions, please do not hesitate to let us know and we will gladly assist you. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

 

Best regards,

Faez_Intel

Intel Customer Support

 

 

 

Mike_Intel
Moderator
232 Views

Hello MisterJ,

 

I hope you're having a wonderful day. I am just sending a soft follow up to check if you already tried the steps in the link that we provided. Please send us an update for us to further assist you if needed. We will wait for your update.

 

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

 

Best regards,

Michael L.

Intel® Customer Support 


IntelSupport
Community Manager
163 Views

Hello MisterJ,

 

Good day and I hope this message finds you well

I am just sending a soft follow up to check if you already tried the steps in the link that we provided. Please send us an update for us to further assist you if needed. We will wait for your update.

 

If you have any questions, please do not hesitate to let us know and we will gladly assist you. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,


Faez_Intel

Intel Customer Support


Faez_Intel
Employee
58 Views

Hi Justin, 

 

Good day and I hope this message finds you well!

 

This is just a follow up regarding the information we requested for us to proceed checking on your query. If you have additional questions or clarifications, feel free to let us know.

 

Should there be no response from you, I’ll make sure to reach out on . Let us know if there are any preferred date/time to reach you so that we can revert accordingly.

 

Best regards,

Faez.

Intel Customer Support 


Faez_Intel
Employee
14 Views

Hi Justin, 

 

Good day and I hope this message finds you well!


This is a another follow up regarding on your issue. As per our previous replies, please help to provide the details/info/ for us to proceed further with the case. 


Should there be no response from you, I’ll make sure to reach out on. Let us know if there are any preferred date/time to reach you so that we can revert accordingly.


Do feel free to inquire us if you have any other questions that you may have and we will gladly assist you accordingly.


Best regards,

 

Faez.

Intel Customer Support 



Reply