X710 - VFs - TX driver issue detected, PF reset issued, when running iperf3

[idata]

Hi,

* In summary:

When I run iperf3 between 2 machines throgh SRIO VFs interface and I got the following error in dmesg:

TX driver issue detected, PF reset issued

* How to setup and detail results

+ I got 2 machines each with 1 X710 4ports 10G cards. These 2 X710 cards has direct cable connect between them.

+ On machine 1, I run the following commands to create VF0 and setup its IP. VF0 has pci address of 0000:04:0a.0

$ echo 1 > /sys/bus/pci/devices/0000\:04\:00.2/sriov_numvfs

$ ifconfig enp4s0f2 mtu 9700

$ ifconfig enp4s10 1.1.1.1 netmask 255.255.255.0 mtu 9216

$ echo 'msg_enable 0xffff' > /sys/kernel/debug/i40e/0000\:04\:00.2/command

+ On machine 2, I run the following commands to create VF0 and setup its IP. VF0 has pci address of 0000:04:0a.0

$ echo 1 > /sys/bus/pci/devices/0000\:04\:00.2/sriov_numvfs

$ ifconfig enp4s0f2 mtu 9700

$ ifconfig enp4s10 1.1.1.2 netmask 255.255.255.0 mtu 9216

+ I was able to ping 1.1.1.2 from machine 1 and ping 1.1.1.1 from machine 2

+ On machine 2, start iperf3 server:

$ iperf3 -s -p 8000

+ On machine 1, start iperf3 client:

$ iperf3 -c 1.1.1.12 -p 8000 -l 64 -p 4

$ dmesg

[ 5895.270158] i40e 0000:04:00.2: Malicious Driver Detection event 0x00 on TX queue 68 PF number 0x02 VF number 0x40

[ 5895.270166] i40e 0000:04:00.2: TX driver issue detected, PF reset issued

[ 5895.270170] i40e 0000:04:00.2: TX driver issue detected on VF 0

[ 5895.270173] i40e 0000:04:00.2: Too many MDD events on VF 0, disabled

[ 5895.270176] i40e 0000:04:00.2: Use PF Control I/F to re-enable the VF

[ 5895.284494] i40evf 0000:04:0a.0: PF reset warning received

[ 5895.284500] i40evf 0000:04:0a.0: Scheduling reset task

[ 5895.334703] i40e 0000:04:00.2: Query for DCB configuration failed, err I40E_ERR_ADMIN_QUEUE_ERROR aq_err I40E_AQ_RC_EPERM

[ 5895.334711] i40e 0000:04:00.2: DCB init failed -53, disabled

[ 5895.612086] i40e 0000:04:00.2: Malicious Driver Detection event 0x00 on TX queue 67 PF number 0x02 VF number 0x40

[ 5895.612096] i40e 0000:04:00.2: Too many MDD events on VF 0, disabled

[ 5895.612098] i40e 0000:04:00.2: Use PF Control I/F to re-enable the VF

[ 5895.640978] i40e 0000:04:00.2: Invalid message from VF 0, opcode 3, len 4

* X710 card ports info

04:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

04:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

04:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

04:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

* This is driver info of PF X710:

driver: i40e

version: 1.4.25-k

firmware-version: 5.04 0x80002549 0.0.0

expansion-rom-version:

bus-info: 0000:04:00.2

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

* This is the PF X710 card info:

04:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

Subsystem: Intel Corporation Ethernet Converged Network Adapter X710

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-

Latency: 0, Cache Line Size: 256 bytes

Interrupt: pin A routed to IRQ 35

Region 0: Memory at dc800000 (64-bit, prefetchable) [size=8M]

Region 3: Memory at dc7f8000 (64-bit, prefetchable) [size=32K]

Expansion ROM at df380000 [disabled] [size=512K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Count=129 Masked-

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00001000

Capabilities: [a0] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+

DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-

MaxPayload 128 bytes, MaxReadReq 512 bytes

DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-

LnkCap: Port # 0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <16us

ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [e0] Vital Product Data

Product Name: XL710 40GbE Controller

Read-only fields:

[PN] Part number:

[EC] Engineering changes:

[FG] Unknown:

[LC] Unknown:

[MN] Manufacture ID:

[PG] Unknown:

[SN] Serial number:

[V0] Vendor specific:

[RV] Reserved: checksum good, 0 byte(s) reserved

Read/write fields:

[V1] Vendor specific:

End

Capabilities: [100 v2] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-

UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+

AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

Capabilities: [140 v1] Device Serial Number 90-9c-a4-ff-ff-fe-fd-3c

Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 3

ARICtl: MFVC- ACS-, Function Group: 0

Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

IOVCap: Migration-, Interrupt Message Number: 000

IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy-

IOVSta: Migration-

Initial VFs: 32, Total VFs: 32, Number of VFs: 1, Function Dependency Link: 02

VF offset: 78, stride: 1, Device ID: 154c

Supported Page Size: 00000553, System Page Size: 00000001

Region 0: Memory at 00000000dc400000 (64-bit, prefetchable)

Region 3: Memory at 00000000dc380000 ...

[idata]

Hi Vtran,

Thank you for posting at Wired Communities. The information log shows the the XL710 firmware version is 5.04, you may try upgrade the firmware to version 5.05. https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-and-X710-Series Please feel free to update me. Thanks,sharon

[idata]

Hi Vtran,

Please let us know if you still additional assistance regarding X710.regards,Vince

[CSmit29 · Accepted solution]

Malicious Driver Detection (MDD) Event - Resolved - New 1.7.11 i40en driver

How is this issue being addressed?

The Malicious Driver Detection issue that we are aware of is addressed in the 1.7.11 i40en driver release for ESXi 6.0, ESXi 6.5 and ESXi 6.7.

Available 1.7.11 i40en driver Download Links: For Intel Ethernet 700 Series Network Adapters ( X710, XL710, XXV710, and X722)

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI60-INTEL-I40EN-1711%26productId%3D564 ESXi 6.0 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI65-INTEL-I40EN-1711%26productId%3D614 ESXi 6.5 i40en 1.7.11 Driver

https://my.vmware.com/web/vmware/details%3FdownloadGroup%3DDT-ESXI67-INTEL-I40EN-1711%26productId%3D742 ESXi 6.7 i40en 1.7.11 Driver

For more information please see the blog post: /community/tech/wired/blog/2018/05/23/malicious-driver-detection-mdd-event-resolved Malicious Driver Detection (MDD) Event – Resolved

[JSang8]

Hi: We are seeing same issue on Linux kernel 4.14 + PF driver 2.7.29 + VF driver 3.6.15 + XL 710 NIC firmware 6.10 and 6.8. We are binding VF drivers to DPDK (18.02) while the PF interfaces are bound to Linux. Can anybody throw light on why Intel firmware is throwing this error? We are running heavy TCP traffic through the system when this happens. Note we are not running on VMWARE.