Intel E810 firmware v3.22

[KristofJaeken]

Hi there,

I'm having trouble with my E810 XXVDA4 adapters. Network links randomly don't work. I have this behavior on several new servers.

I was looking to find firmware version 3.22. There is a mention of it on the vmware compatability guide but I can't seem to find it on the Intel website. Also not yet available from HPE (pci card is in a HPE DL380 server). Latest they have there is v3.10 which I am using now.

Anyone who can help me with this? Much appreciated!

[Caguicla_Intel]

Hello KristofJaeken,

Thank you for posting in Intel Ethernet Communities. 

Please provide the following information that would help in checking your query.

1. When was the issue first encountered?

2. Is there any software of hardware changes prior to issue?

3. How often does the issue occurs?

4. How many adapters are affected by the issue?

5. Are all server connected to the same network?

6. When you mentioned 'Network links randomly don't work', can you share more information regarding this? Do you see any error message on the server? Are the servers having this issue all at the same time?

7. Operating System used

8. Can you share if you see any Yottamark or BradyId stickers from the adapters? This will allow us identify if you are using an Original Equipment Manufacturer(OEM) or retail version of Intel Ethernet Adapter.

Looking forward to your reply. 

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel® Customer Support

[KristofJaeken]

Hi Crisselle,

Thanks for your reply.

The issue was noticed 2 to 3 weeks ago. This is a brand new setup with new servers and new top of rack switches. 22 x HPE Proliant DL380 servers with 2 x E810 XXVDA4 each. Switches are HPE 5945. Connections with 25GB DAC. HPE SPP firmware pack from 2022.03. OS installed is VMware 7.0u3. But SO is not important. Issue is already present when booting into BIOS of the server without any OS loaded. Issue is present on all servers.

Firmware on E810 is version 3.10.

What happens is that sometimes on a server a link is down. Not always the same nic. I reboot and the link may be up, but also not. Sometimes a link starts 'down' and after 10 minutes it changes to up. Changing switch port sometimes helps, but there is no config on switch ports. I also have a case open at HPE for this. Switches don't seem to report any errors.

Ultimate test performed is using a HPE Apollo server which has a Broadcom card. On the same switch ports these work all the time (also no OS installed on this one).

Looks like a firmware issue on the E810 cards to me.

In your point 8. do you mean stickers physically on the PCIe card?

Thanks for your assistance!

[KristofJaeken]

Oh, I forgot. One other thing. On the VMware compatibility guide website there is mention of a firmware version 3.22 for the E810 XXVDA4 adapter but I can't find that anywhere. Is that available for download?

[Caguicla_Intel]

Hello KristofJaeken,

Thank you for the swift response. 

Yes, the stickers are on the physical card. We will be sending an email as we need to identify if you are using an OEM or Retail version of Ethernet card. Please send clear front and back photos of the adapter via email to avoid exposing confidential information. Additionally, please share photos of the optical modules used. 

Regarding your follow up question about Firmware version, we tried looking it on our website and the latest version we had is 3.20. We would suggest contacting VMware support for further assistance. Please visit link below and scroll down to Where do I get technical support? for the instructions on how to contact VMware support. 

Intel® Ethernet Adapters Compatibility with VMware*

https://www.intel.com/content/www/us/en/support/articles/000005693/ethernet-products/700-series-network-adapters-up-to-40gbe.html

Looking forward to your reply. 

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel® Customer Support

[KristofJaeken]

Hi Crisselle,

I sent the photos of the adapter card by email, together with a photo of the DAC cable. Those are HPE JL295A.

Today I updated the firmware on the Intel adapters of 4 servers to firmware version 3.20. After that performed some tests. The issue remains the same. It makes no sense to me. For a nic port that is down after server boot I move the DAC cable to another switch port and link comes up. It would seem something with the switch at first sight. However when I connect the DAC from another server which port was up to the original switch port of the first server, the link comes up as well.

So it's not the switch port nor the server nic, nor the DAC cable. I tried interchanging everything with these 4 servers and there is no clear answer what component is causing the disconnect.

[Caguicla_Intel]

Hello KristofJaeken,

Thank you for providing the information requested.

Upon checking the photos of adapter, we have validated that this is HP version of Original Equipment Manufacturer(OEM) Ethernet adapter. For OEM version of Ethernet adapters, we'd like to set your expectation that we have a limited support it would be best if you can get in touch with HP for further support since they have customized drivers, firmware and software to enable or alter features on your card. They are in best position where you can seek assistance so you don’t lose features or customizations of the Ethernet adapter. The NVM should be coming from the manufacturer and there might be a same issue reported to them.

Here is the list of Computer Manufacturers

https://www.intel.com/content/www/us/en/support/contact-intel.html?tab=system-manufacturers

HP Support

https://support.hp.com/us-en

Feel free to let us know if you have questions or clarifications. 

Awaiting to your reply. 

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello KristofJaeken,

Good day!

This is just a follow up to check if you have additional questions or clarifications on this matter. If you do, please let us know and we will be glad to help. 

We also hope that you are able to contact HP support for further assistance on your request.

Awaiting to hear from you. 

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel Customer Support

[KristofJaeken]

Hello Criselle,

I already was in contact with HPE support before starting this thread since they delivered me the hardware. So thanks for pointing it out, but it's not really helping me. Have any similar cases been reported to Intel regarding the E810 cards?

Anyone else on this forum with similar experience with HPE hardware and E810 cards?

Regards

[Caguicla_Intel]

Hello KristofJaeken,

Thank you for the reply. 

Please allow us to check this request as well with our engineers. However, we'd like to set your expectation that we cannot guarantee anything as this is an OEM card. Rest assured that we will do our best effort support on this one.

We will get back to you as soon as possible but no later than 2-3 business days. 

Hoping for your kind understanding. 

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello KristofJaeken,

Please see below feedback from our engineers. 

Since we confirmed that you are using an HPE E810-XXVDA4 adapter, the Firmware validation should be based on product brand and not on the controller anymore. Based on VMware compatibility guide (link below), 3.1 is the highest version validated for HPE E810-XXVDA4 adapter products. If the Firmware of your E810 is already at version 3.10 and having issues, it may not be Firmware related. With this, we highly suggest to continue seeking support with HP as they have customized drivers and software to enable or alter features on the card. They are in best position where he can seek assistance so he does not lose features or customizations of the Ethernet Adapter. 

VMware Compatibility Guide - I/O Device Search

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=54910&vcl=true

Feel free to let us know if you have additional questions or clarifications. 

Awaiting to your reply. 

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello KristofJaeken,

Good day!

Please let us know if you have additional questions or assistance needed on this request. If yes, kindly advise so we can further check. Otherwise, please confirm if we can now close this request. 

Looking forward to your reply. 

Should there be no response from you, we will follow up after 3 business days.

Best regards,

Crisselle C.

Intel Customer Support 

[KristofJaeken]

Hi Criselle,

Thanks for the reply from the engineers. I'm escalating this at HPE through different channels (support and local sales) and I'll take it up with them. From your point I guess we can call it a day.

Kind regards

Kristof

[paulklevitz]

Hey Kristof,

Long shot, but did you ever hear back from HPE regarding this issue?  We're seeing similar behavior with this card on the latest 4.20 firmware.  Anything you could give me would help out greatly!

Thanks,

Paul

[KristofJaeken]

Hi Paul,

Sure thing!

Ultimately a lot of troubleshooting and digging for documentation was performed by myself, a colleague and a local HP field engineer. The most important configuration triggering the behavior was the FEC setting on the switch ports. We found a setting of "RS-FEC" on each port was what we needed to get the connection stable. FEC is used by 25Gb ethernet. It is also used in fiber channel connections.

Since then we haven't had any issues with connectivity. Even though we did several firmware and driver updates.

Fyi, there is a software tool from Intel that you can install on your esxi host to look into nic configuration in detail. I have used that to track the fec settings. It was called 'Intel-intnetcli_1.6.5.0__esx7.0'. Not sure anymore where you can download it. I still have it in case you want it.

In our case the switches are HPE 5945 with 25Gb DAC cables connected to the DL380 servers.

I really hope this helps or at least sets you in the right direction to fix the issue.

Regards,

Kristof

[Caguicla_Intel]

Hello Kristof,

You're welcome. Appreciate your swift response.

We hope that this issue will be resolved soon with the help of HPE. Please be informed that we will now close this request on our end. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

May you have a good day and stay safe!

Best regards,

Crisselle C.

Intel Customer Support