E810-XXVDA2 fails to link up due to invalid FEC settings - Page 2

MisterJ · ‎01-13-2023

Hi,

System details

I currently have 3 identical setups that all exhibit the same problem. Here is the configuration

OS: ESXi 8.0.0 build 20513097 (latest 8.0a release)
CPU: Intel Xeon E5-2697 v4
Nic: E810-XXVDA2 in a PCIe Gen3 x8 slot
Cables: HPE 845418-B21 QSFP28 breakout to 4x SFP28
PBA: K57775-013
Switch: Celestica DX010 32x100G running SONiC NOS
ESXi Driver: 1.10.5.0-1OEM.702.0.0.17630552
UEFI Driver: 4.0.25
NVM: 4.10
NVM ID: 0x80015188

Problem Statement

I have discovered a few issues with getting this nic to work on ESXi 8.

The biggest issue is the nic does not link up to my switch at 25G speeds. But is capable of linking up at 10G with a different breakout cable. I discovered that the "intnetcli" tool fails when trying to disable "Auto-FEC" mode (which I think is the culprit of the link issue):

[root@hyper-core7:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1       Auto-FEC  No-FEC

 vmnic2       Auto-FEC  No-FEC

[root@hyper-core7:~] esxcli intnet fec set -m "No-FEC" -n vmnic2

Requested FEC mode is not supported. Refer to dmesg for more details.
[root@hyper-core7:~] dmesg | grep icen | tail -4
2023-01-13T18:53:46.313Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.0: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:53:46.314Z cpu26:2097304)icen: indrv_IsDcbEnabled:201: 0000:04:00.1: DCB is enabled, Major version 1, Minor version 1
2023-01-13T18:55:03.942Z cpu13:3489616)WARNING: icen: icen_SetFec:13494: 0000:04:00.1: Requested FEC setting is not supported with the current link topology, Status: VMK_NOT_SUPPORTED
2023-01-13T18:55:03.942Z cpu13:3489616)icen: icen_SetFec:13502: 0000:04:00.1: Supported Fec modes: Auto-FEC
[root@hyper-core7:~]

Workaround

Boot into Windows OS
Confirm links are still down
Use Advanced Networking Properties to disable "Auto-FEC" mode on both ports
Disable and Enable both links, wait for them to come up
Warm reboot into ESXi

When I do this I no longer see "Auto-FEC" mode enabled in "intnetcli" tool on the ESXi hosts and the links stay up:

[root@hyper-core5:~] esxcli intnet fec list

vmnic    Requested FEC  FEC
-------  -------------  -----

 vmnic1         No-FEC  No-FEC

 vmnic2         No-FEC  No-FEC

However, this only works until the system is cold booted. So while this does get the systems up and running it is not ideal and leads me to not want to scale this solution out to more systems (as this would be very tedious on 3 systems let alone a rack full of systems).

Other issues:

The latest version of "intnetcli" does not support ESXi 8 (only 6.7 up to 7.0, doesn't even support 7.1). Intel® ESXCLI Plug-In for Managing Intel® Ethernet Network Adapters
1. I was able to extract the VIB and force install it to bypass the warnings

Open Questions:

Is there a more permanent way to override the "Auto-FEC" mode (with an override in the EEPROM/NVM maybe)?
Is there a way to force set the FEC mode (bypassing the validation)?

Additional Comments:

No SFPs are detected on E810-xxvda4t - Intel Communities: This question was never resolved but the symptoms look very similar.
HPE put out an advisory about this adapter having this exact same issue. However, I am unable to change the adapter FEC setting within ESXi and the option is missing from preboot/HII. Document - Advisory: HPE Network Adapters - Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter for HPE May Not Link With Switches at 25Gb | HPE Support

Faez_Intel · ‎03-08-2023

Hi Justin,

Good day and I hope this message finds you well!

This is just an another follow up on this thread. Please kindly let us know if you have any other assistance needed from this thread and we will assist you accordingly.

Do feel free to inquire us if you have any other questions that you may have and we will gladly assist you accordingly.

Best regards,

Faez.

Intel Customer Support

Faez_Intel · ‎03-13-2023

Hello Justin,

I hope you're having a wonderful day. I am just sending a follow up to check if you still have questions or clarifications. We will wait for your reply. As for now, we will be closing your thread.

If you want to continue support, please reply to this community thread and we will reopen your case or create a new one so that we can continue to support you.

Best regards,

Faez.

Intel Customer Support

tmichaeli · ‎02-23-2024

Hi Faez,

Hi I am from VMware and we are facing link-down issue with Ethernet Controller XXV710 for 25GbE SFP28
after the upgrade to VCF5.1/vSpheer8.0.2

Your intnet plugin in to esxcli doe not work and do not allow me to disable FEC.

I am using Intel Server S2600WF, so there is no UI for the card while booting from BIOS.

Is there way to modify setting on UEFI Shell?

Is there way to disable FEC with some other utility from ESXi shell?

You can find all versions and commands bellow:

root@vcf01-prg-esx01:~] esxcli intnet fec get -n vmnic2

Requested FEC Mode: Auto-FEC FEC Mode: No-FEC

[root@vcf01-prg-esx01:~] esxcli intnet fec set -m "No-FEC" -n vmnic2

ERROR: Unable to get/set info from the driver, Status: 1.

[root@vcf01-prg-esx01:~] esxcli intnet fec list

vmnic Requested FEC FEC

------- ------------- -----

vmnic0 No-FEC No-FEC

vmnic1 No-FEC No-FEC

vmnic2 Auto-FEC No-FEC

vmnic3 Auto-FEC No-FEC

[root@vcf01-prg-esx01:~] esxcli network nic list

Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description

------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------

vmnic0 0000:3d:00.0 i40en Up Up 1000 Full a4:bf:01:83:ba:8f 1500 Intel(R) Ethernet Connection X722 for 10GBASE-T

vmnic1 0000:3d:00.1 i40en Up Up 1000 Full a4:bf:01:83:ba:90 1500 Intel(R) Ethernet Connection X722 for 10GBASE-T

vmnic2 0000:86:00.0 i40en Down Down 0 Half 40:a6:b7:20:a2:10 9000 Intel(R) Ethernet Controller XXV710 for 25GbE SFP28

vmnic3 0000:86:00.1 i40en Up Up 25000 Full 40:a6:b7:20:a2:11 9000 Intel(R) Ethernet Controller XXV710 for 25GbE SFP28

[root@vcf01-prg-esx01:~] esxcli network nic get -n vmnic2

Advertised Auto Negotiation: true

Advertised Link Modes: Auto, 25000BaseCR1/Full

Auto Negotiation: true

Backing DPUId: N/A

Cable Type: DA

Current Message Level: 0

Driver Info:

Bus Info: 0000:86:00:0

Driver: i40en

Firmware Version: 9.40 0x8000ed12 1.3429.0

Version: 2.7.2.0

Link Detected: false

Link Status: Down by linkflapping

Name: vmnic2

PHYAddress: 0

Pause Autonegotiate: false

Pause RX: false

Pause TX: false

Supported Ports: DA

Supports Auto Negotiation: true

Supports Pause: true

Supports Wakeon: false

Transceiver:

Virtual Address: 00:50:56:51:7e:00

Wakeon: None

[root@vcf01-prg-esx01:~] esxcli software vib list | grep intnetcli

intnetcli 1.12.5.0-1OEM.800.1.0.20613240 INT VMwareAccepted 2024-02-22 host

[root@vcf01-prg-esx01:~] esxcli software vib list | grep i40en

i40en 2.7.2.0-1OEM.800.1.0.20613240 INT VMwareCertified 2024-02-21 host

[root@vcf01-prg-esx01:~] vmware -v

VMware ESXi 8.0.2 build-22380479