Intel X710 / i40en driver on vmware - any fix?

[BK6]

I just received a few Dell servers to test as replacements for the vendor we currently use for vsphere. I installed ESXi 6.5u1, patched up for spectre/meltdown so I'm now on build 7967591. The servers are hooked into storage via multiple 10gig ethernet paths handled by dual port X710 NIC's:

[root@vm4:~] esxcli network nic get -n vmnic0

Advertised Auto Negotiation: false

Advertised Link Modes: 10000BaseT/Full

Auto Negotiation: false

Cable Type: DA

Current Message Level: -1

Driver Info:

Bus Info: 0000:18:00:0

Driver: i40en

Firmware Version: 6.00 0x800034e6 18.3.6

Version: 1.3.1

Link Detected: true

Link Status: Up

Name: vmnic0

PHYAddress: 0

Pause Autonegotiate: false

Pause RX: true

Pause TX: true

Supported Ports: DA

Supports Auto Negotiation: false

Supports Pause: true

Supports Wakeon: true

Transceiver:

Virtual Address: 00:50:56:5a:d4:93

Wakeon: MagicPacket(tm)

Almost immediately after putting any real load on them, the NIC simply stops passing traffic. What's worse, it remains in an UP state, so vmware never tries the failover link. Seeing this on the vmware side:

2018-04-01T11:30:41.265Z cpu1:65925)StorageApdHandlerEv: 117: Device or filesystem with identifier [e87ff85e-cb1d8034] has exited the All Paths Down state.

2018-04-01T12:39:19.849Z cpu46:66166)i40en: i40en_HandleMddEvent:6495: Malicious Driver Detection event 0x02 on TX queue 0 PF number 0x00 VF number 0x00

2018-04-01T12:39:19.849Z cpu46:66166)i40en: i40en_HandleMddEvent:6521: TX driver issue detected, PF reset issued

That of course lead me to the following two closed threads in this forum:

https://communities.intel.com/thread/117035 https://communities.intel.com/thread/117035

https://communities.intel.com/thread/117076 https://communities.intel.com/thread/117076

Is it safe to assume this NIC is still broken and will never see fixes, with both sides blaming each other? Can't say it leaves me very happy with Dell either as they knew we were doing vsphere on these, yet they still bundled them.

[BK6]

Following up my own post, there exists newer drivers and firmware for this thing. Unfortunately, the X710 that comes in Dell servers is a proprietary version that Intel firmware tools won't patch.

Latest driver is 1.5.8 for ESXi 6.5:

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-158&productId=614

Installed that, still had the issue shortly after sending traffic across this thing again.

Found the Intel firmware for the X710, currently version 6.01, where the card came with 6.00. Found the later at:

https://downloadmirror.intel.com/25791/eng/XL710_NVMUpdatePackage_v6_01_Linux.tar.gz

Unfortunately, if you try to use that tool, it simply identifies your Dell-branded X710 as "Intel Ethernet 10G 4P X710/I350 rNDC" version 6.00 and update not available. Dell's latest package for it on the 14g server generation, is 18.3.0 which just bundles the 6.00 firmware.

So, appears there is no solution to using this card reliably on vmware.

[idata]

Hi Ispcolohost,

Thank you for posting in Wired Communities and I am sorry to hear what happened. Thank you for sharing the information in case there are other users using the same network card in the same environment.Thanks,Sharon T

[CVibe]

Hi,

Our workaround was to open the console then remove the driver with the command esxcli software vib remove -n i40en

This put it back to the i40e version and this has been stable although the update manager issue is driving us nuts. I have some forums where people said they ripped the nics out completely and replaced them (that is extreme).

Because we had our vCenter server on the host that went down we couldn't bring it back up very easily. We had to remove one of the nic's from the distributed switch and create a standard switch in the same network which we could use to bring it back online and then we switched back to the original network. This sounds easy but we burnt a few hours thinking of alternatives to this.

Anyway i hope this helps.

Thanks

[JGall8]

We also tried to uninstall the drivers but it didn't work...

We only hope that Intel releases a new version or VMware a fix. Our customer is really upset with this situation.

Don't know what else I can do...

Maybe buy a different ones because I don't understand why so many time to fix this issue.

Thanks for your comments, really appreciate.

[idata]

Hi Jrgallego,

Thank you for the information and I am sorry to hear what happened. Let me branch out the post and we will further check.Regards,Sharon T

[idata]

Hi Ispcolohost,

Please be informed Malicious Driver Detection is and open issue for ESX i40en driver version 1.5.8. The Malicious Driver Detection issue that we are aware of will be addressed in the next i40en driver release. The next driver is going through VMware Certification process and expected to release in a couple of weeks. We will have an official statement on this issue by next week. Thank you for your patience on this matter.

Regards,

Sharon T

[CVibe]

Hi,

Anyone have a response to the i40en VIB update that was release on 3rd May?

The patch notes do not provide much info…

http://kb.vmware.com/kb/53080 http://kb.vmware.com/kb/53080

[CSmit29]

/community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved

The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.1 i40en driver release for ESXi 6.7 available for download here: https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI67-INTEL-I40EN-171&productId=745 Link to Download

Note: Drivers that address the MDD issue for ESXi 6.0 & ESXi 6.5 will be available in a future release.

For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved

[BK6]

Has the fix for 6.5 been released yet?

[idata]

Hello Ispcolohost,

Thank you for checking back with us. Has the new driver 1.7.11 from VMware resolved the issue? Please let us know if you still need assistance. If you have any questions please do not hesitate to ask.Best regards,Daniel D

[BK6]

We switched to Mellanox cards which have been rock solid. I do have a test server which is still 710-based though so I'll give it a try with vsphere 6.5 and report back.

[CSmit29]

Hello Ispcolohost,

We appreciate you taking the time to test the driver on your server. We look forward to you sharing your results.

Please let us know if you have any questions.

Thanks,

Intel Ethernet Team.

[DDörf]

The new driver (1.7.11) has been released and fixes the problem. See https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=43711&vcl=true VMware Compatibility Guide - I/O Device Search for a list of drivers.

[CSmit29 · Accepted solution]

Malicious Driver Detection (MDD) Event - Resolved - New 1.7.11 i40en driver

How is this issue being addressed?

The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.11 i40en driver release for ESXi 6.0, ESXi 6.5 and ESXi 6.7.

Available 1.7.11 i40en driver Download Links: For Intel Ethernet 700 Series Network Adapters ( X710, XL710, XXV710, and X722)

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI60-INTEL-I40EN-1711%26productId%3D564 ESXi 6.0 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI65-INTEL-I40EN-1711%26productId%3D614 ESXi 6.5 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI67-INTEL-I40EN-1711%26productId%3D742 ESXi 6.7 i40en 1.7.11 Driver

For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved