Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
5211 Discussions

Intel X710-DA4 / VMware ESXi 6.5u1 - Malicious Driver Detection Event Occured

CKalb
Novice
8,340 Views

Hello,

we're having problems with an Intel X710-DA4 retail card on VMware ESXi 6.5u1. After some time (usually minutes to hours) of sustained traffic on the NIC, we're seeing the following in vmkernel.log:

2017-08-11T12:26:02.554Z cpu18:66233)i40en: i40en_HandleMddEvent:6495: Malicious Driver Detection event 0x02 on TX queue 0 PF number 0x03 VF number 0x00

2017-08-11T12:26:02.554Z cpu18:66233)i40en: i40en_HandleMddEvent:6521: TX driver issue detected, PF reset issued

The network port in question is then apparently shut down, although the link stays up, and it does not pass any more network traffic. Only a reboot of the server will reset the network port and allow traffic to flow through it again.

 

The traffic pattern that leads to that issue usually is TCP traffic of >300MBit/s passing through a firewall virtual machine, entering on one virtual interface and exiting through another.

We are using ESXi 6.5u1 with the built-in i40en driver, as well as the latest NVM firmware version 5.05:

000:82:00.0 8086:1572 8086:0004 vmkernel vmnic2

0000:82:00.1 8086:1572 8086:0000 vmkernel vmnic3

0000:82:00.2 8086:1572 8086:0000 vmkernel vmnic4

0000:82:00.3 8086:1572 8086:0000 vmkernel vmnic5

esxcli network nic get -n vmnic3

Advertised Auto Negotiation: false

Advertised Link Modes: 10000BaseSR/Full

Auto Negotiation: false

Cable Type: FIBRE

Current Message Level: -1

Driver Info:

Bus Info: 0000:82:00:1

Driver: i40en

Firmware Version: 5.05 0x80002898 1.1568.0

Version: 1.3.1

Link Detected: true

Link Status: Up

Name: vmnic3

(...)

More details to curtail the problem:

  • We are not using SR-IOV.
  • The exact driver version is i40en 1.3.1-5vmw.650.1.26.5969303. We have observed the same issue with a previous driver version 1.3.1-1OEM.600.0.0.2768847.
  • The issue happens on multiple hosts, all with the same Intel X710-DA4 adapter.

VMware Support has not been able to resolve the issue for us, saying they have been observing issues with all current X710 drivers and cannot point us into any specific direction - other than asking us to turn to Intel for support.

Honestly, at this point we're at our wits end and do not know how to proceed any further - other than switching to a different manufacturer's network hardware altogether.

Thank you for any helpful advice.

0 Kudos
1 Solution
CSmit29
New Contributor I
5,735 Views

Malicious Driver Detection (MDD) Event - Resolved - New 1.7.11 i40en driver

How is this issue being addressed?

The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.11 i40en driver release for ESXi 6.0, ESXi 6.5 and ESXi 6.7.

Available 1.7.11 i40en driver Download Links: For Intel Ethernet 700 Series Network Adapters ( X710, XL710, XXV710, and X722)

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI60-INTEL-I40EN-1711%26productId%3D564 ESXi 6.0 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI65-INTEL-I40EN-1711%26productId%3D614 ESXi 6.5 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI67-INTEL-I40EN-1711%26productId%3D742 ESXi 6.7 i40en 1.7.11 Driver

For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved

View solution in original post

0 Kudos
13 Replies
idata
Employee
5,735 Views

Hi Nlxck,

 

 

Thank you for posting at Wired Communities. The malicious driver detection is a feature supported by the NIC, please refer to the information at http://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf. This feature monitors queues and VFs for malformed descriptors that might indicate a malicious or buggy driver.

 

 

 

 

Thanks,

 

sharon

 

0 Kudos
CKalb
Novice
5,735 Views

Hi Sharon,

thank you for your explanation. While I realize what malicious driver detection does, we've been using both the 1.3.1-1OEM.600.0.0.2768847 and the 1.3.1-5vmw.650.1.26.5969303 driver versions, both of which cause the same issue of stopping completely legitimate traffic on the network interface.

Another user has been hinting at the fact that this issue has existed for a long time ("I've been dealing with this for 2+ years with no end insight.") in his comment in /thread/117076 https://communities.intel.com/thread/117076. He has also been using another driver version (2.0.6) which, while causing fewer complete stops of traffic, still seems to have the same issue. I have also read similar accounts from other users, both on the Intel Communities and elsewhere.

I have also actually been in contact with Intel Support today (Case ID 02997668). While the person was very friendly, he also could not help me other than telling me that Intel does not support VMware network drivers directly anymore since ESXi 5.x.

Do you have any recommendation on how to proceed any further - other than buying different networking hardware?

Best

CK

0 Kudos
CKalb
Novice
5,736 Views

Any news or any further help from Intel on this topic? Can I provide any more technical details to help resolve these problems?

0 Kudos
idata
Employee
5,736 Views

Hi Nlxck,

 

 

Further checking, as this is a VMWare driver, it is recommended to contact VMware support for further assistance. Thank you for your understanding on this matter.

 

 

Regards,

 

sharon

 

0 Kudos
CKalb
Novice
5,736 Views

Hi Sharon,

as I pointed out in my initial post, I have already been in contact with VMware support: "VMware Support has not been able to resolve the issue for us, saying they have been observing issues with all current X710 drivers and cannot point us into any specific direction - other than asking us to turn to Intel for support." VMware support also pointed out that VMware supposedly does not even have the source code to the drivers.

To sum it up:

I have seen myself forced to order new network hardware from one of your competitors and will desert this ship once it has arrived. Make of that what you will.

Regards,

CK

0 Kudos
idata
Employee
5,736 Views

Hi Nlxck,

 

 

I am pleased to inform you that the issue is addressed in the latest ESX i40en driver version 1.4.3. This driver was signed by VMware already and is expected to be available from VMware VCG site sometime next week.

 

 

Thanks,

 

sharon

 

0 Kudos
idata
Employee
5,736 Views

Hi Nlxck,

 

 

Please feel free to update me if further assistance needed?

 

 

Regards,

 

sharon

 

0 Kudos
MGrei2
Beginner
5,736 Views

Hi Guys

I am facing the same problems.

At the moment I am testing with i40en 1.4.3 on ESXi 6.5U1

Can someone confirm Intels statement, that the driver should address the bug?

regards

Martin

0 Kudos
CKalb
Novice
5,736 Views

We just encountered the Issue again with i40en 1.4.3 and the latest firmware 6.01. So the problem still isn't solved.

0 Kudos
idata
Employee
5,736 Views

Hi NLxck,

 

 

Thank you for the update and I am sorry to hear that. Let me further investigate on this.

 

 

Regards,

 

Sharon

 

0 Kudos
idata
Employee
5,736 Views

Hi NLxck,

 

 

Please provide below information for better investigation:

 

1) vmkernel.log file

 

2) dmesg output

 

3) Details instructions on how to reproduce this issue

 

 

Thanks,

 

Sharon

 

0 Kudos
idata
Employee
5,736 Views

Hi NLxck,

 

 

Please feel free to provide the information requested.

 

 

Thanks,

 

Sharon
0 Kudos
CSmit29
New Contributor I
5,736 Views

Malicious Driver Detection (MDD) Event - Resolved - New 1.7.11 i40en driver

How is this issue being addressed?

The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.11 i40en driver release for ESXi 6.0, ESXi 6.5 and ESXi 6.7.

Available 1.7.11 i40en driver Download Links: For Intel Ethernet 700 Series Network Adapters ( X710, XL710, XXV710, and X722)

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI60-INTEL-I40EN-1711%26productId%3D564 ESXi 6.0 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI65-INTEL-I40EN-1711%26productId%3D614 ESXi 6.5 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI67-INTEL-I40EN-1711%26productId%3D742 ESXi 6.7 i40en 1.7.11 Driver

For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved

0 Kudos
Reply