Intel 810 with VMware ESXI 7.03

[Robbinsj]

Using an Intel 810 25 Gbe adapter with Vmware 7.03 EXI and firmware 3.01 and ESXI driver 1.75

The driver is reporting a lot of pNIC receive errors.   Anyone else seen this issue?  You can see the receive error count at the bottom stats section.   Any idea how to debug/fix this?

ESXI 7.03 build 19193900

esxcli software vib list | grep icen
icen 1.7.5.0-1OEM.702.0.0.17630552 INT VMwareCertified 2022-01-28

esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------
vmnic0 0000:3b:00.0 icen Up Up 25000 Full b4:96:91:9f:d2:c8 9000 Intel(R) Ethernet Controller E810-C for SFP
vmnic1 0000:3b:00.1 icen Up Up 25000 Full b4:96:91:9f:d2:c9 1500 Intel(R) Ethernet Controller E810-C for SFP
vmnic2 0000:3b:00.2 icen Up Up 25000 Full b4:96:91:9f:d2:ca 9000 Intel(R) Ethernet Controller E810-C for SFP
vmnic3 0000:3b:00.3 icen Up Up 25000 Full b4:96:91:9f:d2:cb 1500 Intel(R) Ethernet Controller E810-C for SFP

esxcli network nic get -n vmnic0
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 10000BaseSFI/Full, 25000BaseCR1/Full
Auto Negotiation: true
Cable Type: DA
Current Message Level: 0
Driver Info:
Bus Info: 0000:3b:00:0
Driver: icen
Firmware Version: 3.10 0x8000ad7b 1.3106.0
Version: 1.7.5.0
Link Detected: true
Link Status: Up
Name: vmnic0
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: false
Pause TX: false
Supported Ports: DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver:
Virtual Address: 00:50:56:5d:1e:dd
Wakeon: None

esxcli network nic stats get -n vmnic0
NIC statistics for vmnic0
Packets received: 2596496981
Packets sent: 2343762073
Bytes received: 2295251380679
Bytes sent: 4263038734638
Receive packets dropped: 0
Transmit packets dropped: 0
Multicast packets received: 4322059
Broadcast packets received: 19590472
Multicast packets sent: 214060
Broadcast packets sent: 18864
Total receive errors: 395614546
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0

[shefys]

same thing on 1.7.5.0 (fw 2.00):

esxcli network nic stats get -n vmnic4
NIC statistics for vmnic4
Packets received: 74291750
Packets sent: 22357
Bytes received: 49179469461
Bytes sent: 161358110786
Receive packets dropped: 0
Transmit packets dropped: 0
Multicast packets received: 94568
Broadcast packets received: 9800
Multicast packets sent: 40233343
Broadcast packets sent: 26
Total receive errors: 114498775
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0

[Talexandre]

Hi Robbinsj!

I hope you're doing well!

Mate, normally these events type have a relationship with Upstream device, but tell me the follow infos that you have been used on this topology:

- Upstream Switch device model;

- Cable type / speed;

- FEC Mode setup if you're using 25/100GB;

- SFP type / vendor / model;

Could you please get and share a tcpdump / pcap file btw Switch device Vs Interface NIC? Can you check if these received errors occur without VMs traffic? I mean, only Host ESX and Switch.

Let me know about it please!

Talex

[MikeSully]

Did you happen to resolve this issue? If so, can you please share the solution?

[BelmonteA]

Hi Folks,

Can you share the hardware vendor and model you are using ?  HPE, DELL , CISCO ... etc ????

Does anyone is seeing the issue running ESXi 7.03 on HPE servers ?

If yes,  and you want to investigate the issue please open a case with HPE and share the case number.

** Necessary a valid contract support or valid product warranty with HPE to open a support case **

Best regards.

BelmonteA.

[Maxilib]

Hi man,

Can you resolve your problem ? 

I have same error in all cluster with same nic card...

Br

[PeteK23]

Same issue here:

VMware ESXi, 7.0.3, 20842708

esxcli software vib list | grep icen
icen 1.6.2.0-1 OEM.700.1.0.15843807 INT VMwareCertified 2022-09-02

esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------
vmnic0 0000:04:00.0 ntg3 Up Down 0 Half c8:4b:d6:x:xx:xx 1500 Broadcom Corporation N etXtreme BCM5720 Gigabit Ethernet
vmnic1 0000:04:00.1 ntg3 Up Down 0 Half c8:4b:d6:xx:xx:xx 1500 Broadcom Corporation N etXtreme BCM5720 Gigabit Ethernet
vmnic2 0000:51:00.0 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vmnic3 0000:51:00.1 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vmnic4 0000:8a:00.0 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vmnic5 0000:8a:00.1 icen Up Up 10000 Full b4:96:91:xx:xx:xx 1500 Intel(R) Ethernet Cont roller E810-XXV for SFP
vusb0 Pseudo cdce Up Up 100 Full c8:4b:d6:xx:xx:xx 1500 DellTM iDRAC Virtual N IC USB Device

The TOR switches are Dell S4148F (10GB SFP Switch)

10GB SFP with 10GB DAC cables.

Hardware is Dell VxRail E660F or (PowerEdge R750)

Network Card:  (2) 2 Port 10GB NIC Cards

Intel(R) Ethernet 25G 2P E810-XXV OCP

Intel(R) Ethernet 25G 2P E810-XXV Adapter

Family Firmware Version: 20.5.13

There appears to be a new 21.5.9 Firmware available. 

Found these VMWare Articles:
https://kb.vmware.com/s/article/83627
https://kb.vmware.com/s/article/83243, #This Issue should be fixed in ESX 7.0.3

Dell Support provided some instructions for the
Intel 25G E810 Adapter related to Forward Error Correction (FEC) setting that I will be trying next.

[Ehk]

Hi! Do you have any updates on this? We have the same errors as you.

[PeteK23]

Update:

Found this link related to the same error, https://communities.vmware.com/t5/VMware-vSAN-Discussions/High-pNic-error-rate-detected-Check-the-host-s-vSAN-performance/td-p/2888368

Tried to assign the FEC settings to use 10GB to match the network switch and the Dell 10GB DAC cable, same problem.

Tried to upgrade to the newer VxRail 7.0.241 Software Version (This uses VMware ESXi, 7.0.3, 21313628), same issue.

It appears that HPE servers also have this problem within VMWare with their QLogic FastLinQ NIC and they have posted a firmware/driver fix.

https://support.hpe.com/hpesc/public/docDisplay?docId=a00126305en_us&docLocale=en_US

The HPE release notes show the following:

This product resolves an accounting error related to port statistics and incorrectly recording Rx drops. The incorrect logging of Rx drops can generate a VMware alert message similar to “High pNic error rate detected.;

Here is the link to the Release Notes for the newer Intel e810 firmware version 21.5.9, no accounting fixes mentioned. 

https://dl.dell.com/FOLDER08982528M/2/fw_release_e810.txt?uid=27e63781-f9e1-445b-540f-dc0695b0ab03&fn=fw_release_e810.txt

VMWare's Website also shows the latest Intel NIC driver, but nothing stands out in the Release Notes.

https://customerconnect.vmware.com/downloads/details?downloadGroup=DT-ESXI70-INTEL-ICEN-11130&productId=974&download=true&fileId=219882052778382d916cee052a31953d&uuId=e7799bf4-e722-450f-a1be-d3bf69ff80e6

Version	1.11.3.0
Release Date	2023-02-24