Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4919 Discussions

E810-XXVDA2 fails to link up due to invalid FEC settings

MisterJ
Beginner
10,863 Views

Hi,

 

System details

I currently have 3 identical setups that all exhibit the same problem. Here is the configuration

  • OS: ESXi 8.0.0 build 20513097 (latest 8.0a release)
  • CPU: Intel Xeon E5-2697 v4 
  • Nic: E810-XXVDA2 in a PCIe Gen3 x8 slot
  • Cables: HPE 845418-B21 QSFP28 breakout to 4x SFP28
  • PBA: K57775-013
  • Switch: Celestica DX010 32x100G running SONiC NOS
  • ESXi Driver: 1.10.5.0-1OEM.702.0.0.17630552
  • UEFI Driver: 4.0.25
  • NVM: 4.10
  • NVM ID: 0x80015188

 

Problem Statement

I have discovered a few issues with getting this nic to work on ESXi 8.

 

The biggest issue is the nic does not link up to my switch at 25G speeds. But is capable of linking up at 10G with a different breakout cable. I discovered that the "intnetcli" tool fails when trying to disable "Auto-FEC" mode (which I think is the culprit of the link issue):

 

[root@hyper-core7:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1       Auto-FEC  No-FEC

 vmnic2       Auto-FEC  No-FEC

[root@hyper-core7:~] esxcli intnet fec set -m "No-FEC" -n vmnic2

Requested FEC mode is not supported. Refer to dmesg for more details.
[root@hyper-core7:~] dmesg | grep icen | tail -4
2023-01-13T18:53:46.313Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.0: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:53:46.314Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.1: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:55:03.942Z cpu13:3489616)WARNING: icen: icen_SetFec:13494: 0000:04:00.1: Requested FEC setting is not supported with the current link topology, Status: VMK_NOT_SUPPORTED
2023-01-13T18:55:03.942Z cpu13:3489616)icen: icen_SetFec:13502: 0000:04:00.1: Supported Fec modes: Auto-FEC
[root@hyper-core7:~] 

 

 

Workaround

  1. Boot into Windows OS
  2. Confirm links are still down
  3. Use Advanced Networking Properties to disable "Auto-FEC" mode on both ports
  4. Disable and Enable both links, wait for them to come up
  5. Warm reboot into ESXi

When I do this I no longer see "Auto-FEC" mode enabled in "intnetcli" tool on the ESXi hosts and the links stay up:

 

[root@hyper-core5:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1         No-FEC  No-FEC

 vmnic2         No-FEC  No-FEC

 

However, this only works until the system is cold booted. So while this does get the systems up and running it is not ideal and leads me to not want to scale this solution out to more systems (as this would be very tedious on 3 systems let alone a rack full of systems). 

 

Other issues:

  1. The latest version of "intnetcli" does not support ESXi 8 (only 6.7 up to 7.0, doesn't even support 7.1). Intel® ESXCLI Plug-In for Managing Intel® Ethernet Network Adapters
    1. I was able to extract the VIB and force install it to bypass the warnings

 

Open Questions:

  1. Is there a more permanent way to override the "Auto-FEC" mode (with an override in the EEPROM/NVM maybe)?
  2. Is there a way to force set the FEC mode (bypassing the validation)?

 

Additional Comments:

  1. No SFPs are detected on E810-xxvda4t - Intel Communities: This question was never resolved but the symptoms look very similar. 
  2. HPE put out an advisory about this adapter having this exact same issue. However, I am unable to change the adapter FEC setting within ESXi and the option is missing from preboot/HII. Document - Advisory: HPE Network Adapters - Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter for HPE May Not Link With Switches at 25Gb | HPE Support 

 

0 Kudos
23 Replies
Faez_Intel
Moderator
958 Views

Hi Justin,


Good day and I hope this message finds you well!


This is just an another follow up on this thread. Please kindly let us know if you have any other assistance needed from this thread and we will assist you accordingly.


Do feel free to inquire us if you have any other questions that you may have and we will gladly assist you accordingly.


Best regards,

Faez.

Intel Customer Support


0 Kudos
Faez_Intel
Moderator
948 Views

Hello Justin,

 

I hope you're having a wonderful day. I am just sending a follow up to check if you still have questions or clarifications. We will wait for your reply. As for now, we will be closing your thread.

 

If you want to continue support, please reply to this community thread and we will reopen your case or create a new one so that we can continue to support you.


Best regards,

 

Faez.

Intel Customer Support


0 Kudos
tmichaeli
Beginner
383 Views

Hi Faez,

Hi I am from VMware and we are facing link-down issue with Ethernet Controller XXV710 for 25GbE SFP28 
after the upgrade to VCF5.1/vSpheer8.0.2 

Your intnet plugin in to esxcli doe not work and do not allow me to disable FEC.

I am using Intel Server S2600WF, so there is no UI for the card while booting from BIOS.

Is there way to modify setting on UEFI Shell?

Is there way to disable FEC with some other utility from ESXi shell?

You can find all versions and commands bellow:

root@vcf01-prg-esx01:~] esxcli intnet fec get -n vmnic2

Requested FEC Mode: Auto-FEC FEC Mode: No-FEC
[root@vcf01-prg-esx01:~] esxcli intnet fec set -m "No-FEC" -n vmnic2

ERROR: Unable to get/set info from the driver, Status: 1.

[root@vcf01-prg-esx01:~] esxcli intnet fec list

vmnic Requested FEC FEC
------- ------------- -----

vmnic0 No-FEC No-FEC

vmnic1 No-FEC No-FEC

vmnic2 Auto-FEC No-FEC

vmnic3 Auto-FEC No-FEC

[root@vcf01-prg-esx01:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------
vmnic0 0000:3d:00.0 i40en Up Up 1000 Full a4:bf:01:83:ba:8f 1500 Intel(R) Ethernet Connection X722 for 10GBASE-T
vmnic1 0000:3d:00.1 i40en Up Up 1000 Full a4:bf:01:83:ba:90 1500 Intel(R) Ethernet Connection X722 for 10GBASE-T
vmnic2 0000:86:00.0 i40en Down Down 0 Half 40:a6:b7:20:a2:10 9000 Intel(R) Ethernet Controller XXV710 for 25GbE SFP28
vmnic3 0000:86:00.1 i40en Up Up 25000 Full 40:a6:b7:20:a2:11 9000 Intel(R) Ethernet Controller XXV710 for 25GbE SFP28

[root@vcf01-prg-esx01:~] esxcli network nic get -n vmnic2
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 25000BaseCR1/Full
Auto Negotiation: true
Backing DPUId: N/A
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:86:00:0
Driver: i40en
Firmware Version: 9.40 0x8000ed12 1.3429.0
Version: 2.7.2.0
Link Detected: false
Link Status: Down by linkflapping
Name: vmnic2
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:51:7e:00
Wakeon: None


[root@vcf01-prg-esx01:~] esxcli software vib list | grep intnetcli
intnetcli 1.12.5.0-1OEM.800.1.0.20613240 INT VMwareAccepted 2024-02-22 host
[root@vcf01-prg-esx01:~] esxcli software vib list | grep i40en
i40en 2.7.2.0-1OEM.800.1.0.20613240 INT VMwareCertified 2024-02-21 host
[root@vcf01-prg-esx01:~] vmware -v
VMware ESXi 8.0.2 build-22380479
 
0 Kudos
Reply