- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
we're having problems with an Intel X710-DA4 retail card on VMware ESXi 6.5u1. After some time (usually minutes to hours) of sustained traffic on the NIC, we're seeing the following in vmkernel.log:
2017-08-11T12:26:02.554Z cpu18:66233)i40en: i40en_HandleMddEvent:6495: Malicious Driver Detection event 0x02 on TX queue 0 PF number 0x03 VF number 0x00
2017-08-11T12:26:02.554Z cpu18:66233)i40en: i40en_HandleMddEvent:6521: TX driver issue detected, PF reset issued
The network port in question is then apparently shut down, although the link stays up, and it does not pass any more network traffic. Only a reboot of the server will reset the network port and allow traffic to flow through it again.
The traffic pattern that leads to that issue usually is TCP traffic of >300MBit/s passing through a firewall virtual machine, entering on one virtual interface and exiting through another.
We are using ESXi 6.5u1 with the built-in i40en driver, as well as the latest NVM firmware version 5.05:
000:82:00.0 8086:1572 8086:0004 vmkernel vmnic2
0000:82:00.1 8086:1572 8086:0000 vmkernel vmnic3
0000:82:00.2 8086:1572 8086:0000 vmkernel vmnic4
0000:82:00.3 8086:1572 8086:0000 vmkernel vmnic5
esxcli network nic get -n vmnic3
Advertised Auto Negotiation: false
Advertised Link Modes: 10000BaseSR/Full
Auto Negotiation: false
Cable Type: FIBRE
Current Message Level: -1
Driver Info:
Bus Info: 0000:82:00:1
Driver: i40en
Firmware Version: 5.05 0x80002898 1.1568.0
Version: 1.3.1
Link Detected: true
Link Status: Up
Name: vmnic3
(...)
More details to curtail the problem:
- We are not using SR-IOV.
- The exact driver version is i40en 1.3.1-5vmw.650.1.26.5969303. We have observed the same issue with a previous driver version 1.3.1-1OEM.600.0.0.2768847.
- The issue happens on multiple hosts, all with the same Intel X710-DA4 adapter.
VMware Support has not been able to resolve the issue for us, saying they have been observing issues with all current X710 drivers and cannot point us into any specific direction - other than asking us to turn to Intel for support.
Honestly, at this point we're at our wits end and do not know how to proceed any further - other than switching to a different manufacturer's network hardware altogether.
Thank you for any helpful advice.
- Tags:
- VMware
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Malicious Driver Detection (MDD) Event - Resolved - New 1.7.11 i40en driver
How is this issue being addressed?
The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.11 i40en driver release for ESXi 6.0, ESXi 6.5 and ESXi 6.7.
Available 1.7.11 i40en driver Download Links: For Intel Ethernet 700 Series Network Adapters ( X710, XL710, XXV710, and X722)
https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI60-INTEL-I40EN-1711%26productId%3D564 ESXi 6.0 i40en 1.7.11 Driver
https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI65-INTEL-I40EN-1711%26productId%3D614 ESXi 6.5 i40en 1.7.11 Driver
https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI67-INTEL-I40EN-1711%26productId%3D742 ESXi 6.7 i40en 1.7.11 Driver
For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Nlxck,
Thank you for posting at Wired Communities. The malicious driver detection is a feature supported by the NIC, please refer to the information at http://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdf. This feature monitors queues and VFs for malformed descriptors that might indicate a malicious or buggy driver.
Thanks,
sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sharon,
thank you for your explanation. While I realize what malicious driver detection does, we've been using both the 1.3.1-1OEM.600.0.0.2768847 and the 1.3.1-5vmw.650.1.26.5969303 driver versions, both of which cause the same issue of stopping completely legitimate traffic on the network interface.
Another user has been hinting at the fact that this issue has existed for a long time ("I've been dealing with this for 2+ years with no end insight.") in his comment in /thread/117076 https://communities.intel.com/thread/117076. He has also been using another driver version (2.0.6) which, while causing fewer complete stops of traffic, still seems to have the same issue. I have also read similar accounts from other users, both on the Intel Communities and elsewhere.
I have also actually been in contact with Intel Support today (Case ID 02997668). While the person was very friendly, he also could not help me other than telling me that Intel does not support VMware network drivers directly anymore since ESXi 5.x.
Do you have any recommendation on how to proceed any further - other than buying different networking hardware?
Best
CK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any news or any further help from Intel on this topic? Can I provide any more technical details to help resolve these problems?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Nlxck,
Further checking, as this is a VMWare driver, it is recommended to contact VMware support for further assistance. Thank you for your understanding on this matter.
Regards,
sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sharon,
as I pointed out in my initial post, I have already been in contact with VMware support: "VMware Support has not been able to resolve the issue for us, saying they have been observing issues with all current X710 drivers and cannot point us into any specific direction - other than asking us to turn to Intel for support." VMware support also pointed out that VMware supposedly does not even have the source code to the drivers.
To sum it up:
- All of the X(L)710 firmware/driver combinations on the VMware HCL (https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=37980 VMware Compatibility Guide - I/O Device Search) seem to have issues in one way or another that prohibit stable production use of your product with ESX(i), esp. in SDN scenarios.
- Neither VMware nor Intel want to take the blame and/or initiate any form of collaborative process between VMware and Intel that might be suitable to fix the problem.
- If you do just a little bit of research, you will notice multiple of both respective companies' customers having similar, sustained issues, e.g.:
- https://communities.vmware.com/thread/552470 Install ESXi 6.5 on R730 with Intel X710 |VMware Communities
- Meanwhile, both VMware and Intel both heap great praise on their partnership, which - from our perspective - only seems to exist on paper:
- https://www.vmware.com/partners/global-alliances/intel.html Intel and VMware Virtualization Solutions
- https://www.intel.com/content/www/us/en/cloud-computing/intel-and-vmware-partnership.html Intel and VMware Partnership
I have seen myself forced to order new network hardware from one of your competitors and will desert this ship once it has arrived. Make of that what you will.
Regards,
CK
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Nlxck,
I am pleased to inform you that the issue is addressed in the latest ESX i40en driver version 1.4.3. This driver was signed by VMware already and is expected to be available from VMware VCG site sometime next week.
Thanks,
sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Nlxck,
Please feel free to update me if further assistance needed?
Regards,
sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Guys
I am facing the same problems.
At the moment I am testing with i40en 1.4.3 on ESXi 6.5U1
Can someone confirm Intels statement, that the driver should address the bug?
regards
Martin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We just encountered the Issue again with i40en 1.4.3 and the latest firmware 6.01. So the problem still isn't solved.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi NLxck,
Thank you for the update and I am sorry to hear that. Let me further investigate on this.
Regards,
Sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi NLxck,
Please provide below information for better investigation:
1) vmkernel.log file
2) dmesg output
3) Details instructions on how to reproduce this issue
Thanks,
Sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi NLxck,
Please feel free to provide the information requested.
Thanks,
Sharon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Malicious Driver Detection (MDD) Event - Resolved - New 1.7.11 i40en driver
How is this issue being addressed?
The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.11 i40en driver release for ESXi 6.0, ESXi 6.5 and ESXi 6.7.
Available 1.7.11 i40en driver Download Links: For Intel Ethernet 700 Series Network Adapters ( X710, XL710, XXV710, and X722)
https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI60-INTEL-I40EN-1711%26productId%3D564 ESXi 6.0 i40en 1.7.11 Driver
https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI65-INTEL-I40EN-1711%26productId%3D614 ESXi 6.5 i40en 1.7.11 Driver
https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI67-INTEL-I40EN-1711%26productId%3D742 ESXi 6.7 i40en 1.7.11 Driver
For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page