Intel X710 on Huawei server-malicious driver detection appear in the log

[JGall8]

Hello,

We have the same issue. We have 30 Huawei servers with Intel X710 card and vsphere 6.5 u1 patch2. VM network connections on a Distributed Virtual Switch seem to freeze after this error appears in vmware logs:

"2018-03-30T07:35:50.216Z cpu5:66200)i40en: i40en_HandleMddEvent:6495: Malicious Driver Detection event 0x02 on TX queue 0 PF number 0x00 VF number 0x00

2018-03-30T07:35:50.216Z cpu5:66200)i40en: i40en_HandleMddEvent:6521: TX driver issue detected, PF reset issued

2018-03-30T07:36:20.213Z cpu48:70776)WARNING: NetPort: 1934: failed to disable port 0x300000d on DvsPortset-0: Busy

2018-03-30T07:36:20.213Z cpu48:70776)netschedHClk: NetSchedHClkPortQuiesce:4918: vmnic1: received a force quiesce for port 0x300000d

2018-03-30T07:36:20.213Z cpu48:70776)netschedHClk: NetSchedHClkHashQuiesceHierarchyIter:396: vmnic1: dropped 501 pkts from queue netsched.pools.vm.50331661 while quiescing port 0x300000d"

After this, multiple Dvs ports fail and packets are dropped (vmnic1: dropped 501 pkts from queue).

This has happened on 3 ESXi servers already, causing downtime on VMs.

The logs show the same sequence of events on all 3 servers:

1. Malicious Driver Detection event
2. failed to disable port ___ on DvsPortset-0: Busy
3. vmnicx: dropped ___ pkts from queue

Only way to recover the VM networking is to vMotion to a different host.

I am attaching the vmkernel log file for this server. Issues start at 2018-03-30T07:35:50.216Z

We are really DESPERATED... We are working in big migration project and we are totally frozen!!!

We have tried to installed the last drivers "1.5.8" and the issue happends again.

NEED URGENT HELP!!!

[idata]

Hi Jrgallego,

Thank you for the post and I am sorry to hear what happened. Please share where did you download the driver for the X710 network card? Please provide the link.Regards,Sharon T

[BK6]

Driver link: https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-158&productId=614 https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-158&productId=614https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-158&productId=614 https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-158&productId=614

[JGall8]

Yes. I downloaded from VMware site:

" https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-I40EN-158&productId=614"

Thanks in advance.

[idata]

Hi Jrgallego,

Thank you for the information. Are these X710 network card Intel retail unit or OEM model?

For Intel retail version, you can check Board # (PBA) at this website https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-i-o/ethernet-products.html List of Supported Retail Intel® Ethernet Adapters

If the NICs are OEM, they will have different Board# (PBA).

Or you can follow the Identify your adapter PCI Device ID in the website to check the Vendor and Device ID.

Regards,

Sharon T

[idata]

Hi Jrgallego,

We will be also checking here in case we have information to share. Thanks.Regards,Sharon T

[JGall8]

Really appreciate your comments...

Anyway when have you planned to release a new version for this driver that works with vsphere 6.5 U1?

Or do you have no notice of the issues of this card with another customers since several months ago?

Best regards.

[idata]

Hi Jrgallego,

Thank you for the reply. I don't have information about the driver release, I need to double check. Regards,Sharon T

[WAL]

I am also experiencing the same issue with the driver putting the port into a failed mode (causing ESX top drop 100% of the packets on that nic)

ESXi 6.5 U1, 7967591

2018-04-20T17:50:11.627Z cpu20:66324)i40en: i40en_HandleMddEvent:6969: Malicious Driver Detection event 0x02 on TX queue 0 PF number 0x01 VF number 0x00

2018-04-20T17:50:11.627Z cpu20:66324)i40en: i40en_HandleMddEvent:6995: TX driver issue detected, PF reset issued

We have also tried different HP firmware and driver levels. We are using the HP 562SFP+ and 562FLR-SFP+ cards (based on the Intel X710 chip).

Digging around I found this in the release notes from the Intel driver package:

In a virtualized environment, on Intel(R) Server Adapters that support SR-IOV,

the virtual function (VF) may be subject to malicious behavior.

Software-generated frames are not expected and can throttle traffic between the

host and the virtual switch, reducing performance. To resolve this issue,

configure all SR-IOV enabled ports for VLAN tagging. This configuration allows

unexpected, and potentially malicious, frames to be dropped.

I do have some options on our HP DL380 Gen10 servers that may fix the driver failing but we will not be able to additional testing for a few days at least. I wanted to see if you have tried either enabling VLAN support for the NIC in the UEFI/BIOS or turning off SR-IOV entirely for the NIC/SYSTEM from the UEFI/BIOS?

[idata]

Hi Jrgallego,

Further checking, the information I received is Malicious Driver Detection is and open issue for ESX i40en driver version 1.5.8. The Malicious Driver Detection issue that we are aware of will be addressed in the next i40en driver release. The next driver is going through VMware Certification process and expected to release in a couple of weeks. We will have an official statement on this issue by next week. Thank you for your patience on this matter.

Regards,

Sharon T

[JGall8]

Hello. So sorry I didn't know that information untill now...

Thanks for share it.

Regards.

[JGall8]

Really appreciate your answer and your work.

Let's hope the driver come in the next days.

Thanks in advance.

[idata]

Hi Jrgallego,

Please refer to our latest update about the Malicious Driver Detection Event at https://communities.intel.com/community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved https://communities.intel.com/community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Thanks,Sharon T

[JGall8]

Hi. Do you know if this new driver is only for VMware 6.7 or I can install in my servers with 6.5?

Regards.

[JGall8]

My issue is with vSphere 6.5. Can I install this new driver in this version?

Need to know urgent please!!!

[BK6]

The notice states:

Note: Drivers that address the MDD issue for ESXi 6.0 & ESXi 6.5 will be available in a future release.

[JGall8]

Hello,

Could you tell us when is it suposed to be released the new version for 6.5 please???

Thanks.

[idata]

Hi Jrgallego,

Thank you for the follow up. Please be informed there is no information about the release date. you may monitor this thread for the updates.https://communities.intel.com/community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved https://communities.intel.com/community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolvedRegards,Sharon TIntel Customer SupportAgent under contract to Intel