Intel 810 with VMware ESXI 7.03

[Robbinsj]

Using an Intel 810 25 Gbe adapter with Vmware 7.03 EXI and firmware 3.01 and ESXI driver 1.75

The driver is reporting a lot of pNIC receive errors.   Anyone else seen this issue?  You can see the receive error count at the bottom stats section.   Any idea how to debug/fix this?

ESXI 7.03 build 19193900

esxcli software vib list | grep icen
icen 1.7.5.0-1OEM.702.0.0.17630552 INT VMwareCertified 2022-01-28

esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------
vmnic0 0000:3b:00.0 icen Up Up 25000 Full b4:96:91:9f:d2:c8 9000 Intel(R) Ethernet Controller E810-C for SFP
vmnic1 0000:3b:00.1 icen Up Up 25000 Full b4:96:91:9f:d2:c9 1500 Intel(R) Ethernet Controller E810-C for SFP
vmnic2 0000:3b:00.2 icen Up Up 25000 Full b4:96:91:9f:d2:ca 9000 Intel(R) Ethernet Controller E810-C for SFP
vmnic3 0000:3b:00.3 icen Up Up 25000 Full b4:96:91:9f:d2:cb 1500 Intel(R) Ethernet Controller E810-C for SFP

esxcli network nic get -n vmnic0
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 10000BaseSFI/Full, 25000BaseCR1/Full
Auto Negotiation: true
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:3b:00:0
Driver: icen
Firmware Version: 3.10 0x8000ad7b 1.3106.0
Version: 1.7.5.0
Link Detected: true
Link Status: Up
Name: vmnic0
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:5d:1e:dd
Wakeon: None

esxcli network nic stats get -n vmnic0
NIC statistics for vmnic0
Packets received: 2596496981
Packets sent: 2343762073
Bytes received: 2295251380679
Bytes sent: 4263038734638
Receive packets dropped: 0
Transmit packets dropped: 0
Multicast packets received: 4322059
Broadcast packets received: 19590472
Multicast packets sent: 214060
Broadcast packets sent: 18864
Total receive errors: 395614546
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0

[shefys]

same thing on 1.7.5.0 (fw 2.00):

esxcli network nic stats get -n vmnic4
NIC statistics for vmnic4
Packets received: 74291750
Packets sent: 22357
Bytes received: 49179469461
Bytes sent: 161358110786
Receive packets dropped: 0
Transmit packets dropped: 0
Multicast packets received: 94568
Broadcast packets received: 9800
Multicast packets sent: 40233343
Broadcast packets sent: 26
Total receive errors: 114498775
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0

[Talexandre]

Hi Robbinsj!

I hope you're doing well!

Mate, normally these events type have a relationship with Upstream device, but tell me the follow infos that you have been used on this topology:

- Upstream Switch device model;

- Cable type / speed;

- FEC Mode setup if you're using 25/100GB;

- SFP type / vendor / model;

Could you please get and share a tcpdump / pcap file btw Switch device Vs Interface NIC? Can you check if these received errors occur without VMs traffic? I mean, only Host ESX and Switch.

Let me know about it please!

Talex

[MikeSully]

Did you happen to resolve this issue? If so, can you please share the solution?

[BelmonteA]

Hi Folks,

Can you share the hardware vendor and model you are using ?  HPE, DELL , CISCO ... etc ????

Does anyone is seeing the issue running ESXi 7.03 on HPE servers ?

If yes,  and you want to investigate the issue please open a case with HPE and share the case number.

** Necessary a valid contract support or valid product warranty with HPE to open a support case **

Best regards.

BelmonteA.

[Maxilib]

Hi man,

Can you resolve your problem ? 

I have same error in all cluster with same nic card...

Br

[PeteK23]

Same issue here:

VMware ESXi, 7.0.3, 20842708

esxcli software vib list | grep icen
icen 1.6.2.0-1 OEM.700.1.0.15843807 INT VMwareCertified 2022-09-02

esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------
vmnic0 0000:04:00.0 ntg3 Up Down 0 Half c8:4b:d6:x:xx:xx 1500 Broadcom Corporation N etXtreme BCM5720 Gigabit Ethernet
vmnic1 0000:04:00.1 ntg3 Up Down 0 Half c8:4b:d6:xx:xx:xx 1500 Broadcom Corporation N etXtreme BCM5720 Gigabit Ethernet
vmnic2 0000:51:00.0 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vmnic3 0000:51:00.1 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vmnic4 0000:8a:00.0 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vmnic5 0000:8a:00.1 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vusb0 Pseudo cdce Up Up 100 Full c8:4b:d6:xx:xx:xx 1500 DellTM iDRAC Virtual N IC USB Device

The TOR switches are Dell S4148F (10GB SFP Switch)

10GB SFP with 10GB DAC cables.

Hardware is Dell VxRail E660F or (PowerEdge R750)

Network Card:  (2) 2 Port 10GB NIC Cards

Intel(R) Ethernet 25G 2P E810-XXV OCP

Intel(R) Ethernet 25G 2P E810-XXV Adapter

Family Firmware Version: 20.5.13

There appears to be a new 21.5.9 Firmware available. 

Found these VMWare Articles:
https://kb.vmware.com/s/article/83627
https://kb.vmware.com/s/article/83243, #This Issue should be fixed in ESX 7.0.3

Dell Support provided some instructions for the
Intel 25G E810 Adapter related to Forward Error Correction (FEC) setting that I will be trying next.

[Ehk]

Hi! Do you have any updates on this? We have the same errors as you.

[PeteK23]

Update:

Found this link related to the same error, https://communities.vmware.com/t5/VMware-vSAN-Discussions/High-pNic-error-rate-detected-Check-the-host-s-vSAN-performance/td-p/2888368

Tried to assign the FEC settings to use 10GB to match the network switch and the Dell 10GB DAC cable, same problem.

Tried to upgrade to the newer VxRail 7.0.241 Software Version (This uses VMware ESXi, 7.0.3, 21313628), same issue.

It appears that HPE servers also have this problem within VMWare with their QLogic FastLinQ NIC and they have posted a firmware/driver fix.

https://support.hpe.com/hpesc/public/docDisplay?docId=a00126305en_us&docLocale=en_US

The HPE release notes show the following:

This product resolves an accounting error related to port statistics and incorrectly recording Rx drops. The incorrect logging of Rx drops can generate a VMware alert message similar to “High pNic error rate detected.;

Here is the link to the Release Notes for the newer Intel e810 firmware version 21.5.9, no accounting fixes mentioned. 

https://dl.dell.com/FOLDER08982528M/2/fw_release_e810.txt?uid=27e63781-f9e1-445b-540f-dc0695b0ab03&fn=fw_release_e810.txt

VMWare's Website also shows the latest Intel NIC driver, but nothing stands out in the Release Notes.

https://customerconnect.vmware.com/downloads/details?downloadGroup=DT-ESXI70-INTEL-ICEN-11130&productId=974&download=true&fileId=219882052778382d916cee052a31953d&uuId=e7799bf4-e722-450f-a1be-d3bf69ff80e6

Version	1.11.3.0
Release Date	2023-02-24

[tempestv8]

Am seeing the same thing with an Intel E810 OEM NIC on a HPE Proliant DL380 Gen10 server.

NICs were previously on ICEN driver version 1.7.4 with HPE firmware 3.00 and receiving the error message "High pNic error rate detected. Check the host's vSAN performance view for details" - happening to both servers in a vSAN cluster.

Upgraded the NICs to ICEN driver version 1.8.4.0 with HPE firmware 3.20 (latest available currently) and still receiving the same error message.

[BelmonteA]

PROBLEM IDENTIFIED AND NEW DEVICE DRIVER IS UNDER DEVELOPMENT -  POSTED ON JULY 05, 2023.

Hi Folks,
The "high pNIC receive errors"  was identified by Intel, the device driver is reporting false checksum errors.
Saying that, this is a cosmetic issue and there is no risk to the production or performance.
Intel is working in a new device driver, next version will include the fix.
I have a support case opened with HPE, Support team reported that HPE will include the fix in the next driver package HPE SPP in October 2023.
It can be more or less time depending of hardware vendor you are working with.
regards
Belmonte

[BelmonteA]

PROBLEM IDENTIFIED AND NEW DEVICE DRIVER IS UNDER DEVELOPMENT -  POSTED ON JULY 05, 2023.

Hi Folks,
The "high pNIC receive errors"  was identified by Intel, the device driver is reporting false checksum errors.
Saying that, this is a cosmetic issue and there is no risk to the production or performance.
Intel is working in a new device driver, next version will include the fix.
I have a support case opened with HPE, Support team reported that HPE will include the fix in the next driver package HPE SPP in October 2023.
It can be more or less time depending of hardware vendor you are working with.
regards
Belmonte

[tempestv8]

A new driver for the HPE Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter has been released, along with new firmware.

Driver (version 2023.09.00) can be downloaded from here: https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_1d26406a274c49bca4a792d028

Firmware (version 4.20) can be downloaded from here: https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_8dec9b5d239f4e5ca376e1076e

Hopefully this fixes the cosmetic issue.  Please test and reply back with findings.

[tempestv8]

The latest driver and firmware above, which came from the current latest Service Pack for ProLiant version 2023.09.00.00, does NOT resolve the issue.  Immediately after applying the latest firmware and icen driver onto a server equipped with the HPE Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter, the high pnic receive errors continue to appear.

Kernel is Vmware ESXi 7.0 U3N

[BelmonteA]

The driver was not released yet in the HPE website or SPP , the device driver above is from jul 19, 2023 .. you have installed the old driver that does not include the fix.

the new driver is ready and added into the SPP but the adapter FW is not ready, HPE still testing it , is almost there but the NEW SPP is not available for public.  

Meanwhile you can download the driver from VMware website.

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=53345&deviceCategory=io&details=1&partner=46&releases=578&keyword=E810&deviceTypes=6&page=2&display_interval=10&sortColumn=Partner&sortOrder=Asc

Version 1.12.5 or higher has the fix included.

driver ->>> icen version 1.12.5.0-1OEM

Firmware ->> 4.30

Thanks.

Belmontea

[tempestv8]

HPE has released Service Pack for Proliant 2024.04.00.00.

In this SPP, there is an updated Intel icen driver for Vmware ESXi 7.0 which addresses the high pNIC error messages.  This is the driver for the Intel E810-XXVDA2 Ethernet 10/25Gb 2-port SFP28 Adapter for HPE, p/n P08443-B21

The ESXi 7.0 driver (version 2023.10.00), can be downloaded directly from this link:  https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_1d26406a274c49bca4a792d028

The accompanying firmware (version 4.30) can be downloaded directly from this link:  https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_af3245b2ddf14a42b9731f1f7a