Detect input errors with XL710-QDA2

[TISHI17]

Hi,

I have a problem with Intel XL710-QDA2. When data is transmitted from XL710-QDA2, it detects switch input error.

*sample*

--- show int ---

RX

27997990 unicast packets 14146 multicast packets 36 broadcast packets

28012172 input packets 41674865640 bytes

27161182 jumbo packets 0 storm suppression packets

0 runts 0 giants 168 CRC 0 no buffer

336 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

--- end ---

Although errors are not detected in the test using iperf, they are sporadically detected in storage vmotion and iscsi-traffic of ctld.

The conditions are described below.

OS: FreeBSD 11.1R, ESXi 6.0

NIC: Intel XL710-QDA2

DAC: AVAYA AA1404032-E6, Cisco QSFP-H40G-AOC15M

driver: ESXi latest(2.0.7, nvm 6.01), FreeBSD latest(1.9.7, nvm 6.01)

switch: Cisco Nexus3232C

(*) It does not occur in ConnectX4 or X5 of Mellanox.

I replaced the cable and upgraded the OS of the switch, but it was not fixed.

Please tell me how to solve this problem.

[idata]

Hi Takunii,

Thank you for posting in Wired Communities. I can see you are using Cisco QSFP-H40G-AOC15M cable, please refer to the website below for the supported cable for XL710 series network adapter, it is recommended to use the validated cables: https://www.intel.com/content/www/us/en/support/articles/000006051/network-and-i-o/ethernet-products.htmlThanks,Sharon T

[TISHI17]

Hi Sharon,

Some of the same cards do not cause errors. I examined it and found the next clear difference.

• Problematic NIC

# qcu /devices

Intel(R) QSFP+ Configuration Utility

QCU version: v2.30.16.08

Copyright(C) 2014 - 2017 by Intel Corporation.

Software released under Intel Proprietary License.

NIC Seg:Bus Ven-Dev Mode Adapter Name

=== ======= ========= ======== =================================================

1) 000:136 8086-1583 Invalid Intel(R) Ethernet Converged Network Adapter XL710

# qcu /nic=1 /info

Intel(R) QSFP+ Configuration Utility

QCU version: v2.30.16.08

Copyright(C) 2014 - 2017 by Intel Corporation.

Software released under Intel Proprietary License.

Adapter supports QSFP+ Configuration modification.

Current Configuration: Invalid

Supported Configurations:

4x10

2x2x10A (Top 2 lanes on Module 0; Bottom 2 lanes on Module 1)

2x2x10B (Bottom 2 lanes on both modules)

• Normal NIC

# qcu /devices

Intel(R) QSFP+ Configuration Utility

QCU version: v2.30.16.08

Copyright(C) 2014 - 2017 by Intel Corporation.

Software released under Intel Proprietary License.

NIC Seg:Bus Ven-Dev Mode Adapter Name

=== ======= ========= ======== =================================================

1) 000:133 8086-1583 2x40 Intel(R) Ethernet Converged Network Adapter XL710

# qcu /nic=1 /info

Intel(R) QSFP+ Configuration Utility

QCU version: v2.30.16.08

Copyright(C) 2014 - 2017 by Intel Corporation.

Software released under Intel Proprietary License.

Adapter supports QSFP+ Configuration modification.

Current Configuration: 2x40

Supported Configurations:

1x40

2x40

4x10

2x2x10A (Top 2 lanes on Module 0; Bottom 2 lanes on Module 1)

2x2x10B (Bottom 2 lanes on both modules)

It seems that there is a problem with the mode when confirming the above difference.

It seems that it does not support 2x40, but is it possible to fix it?

Regards,

Takunii

[idata]

Hi Takunii,

Thank you for sharing additional information. Let me further check.Regards,Sharon T

[idata]

Hi Takunii,

Based on the QCU, the XL710-QDA2 with issue supports the following configuration mode:4x102x2x10A (Top 2 lanes on Module 0; Bottom 2 lanes on Module 1)2x2x10B (Bottom 2 lanes on both modules) While the working XL710-QDA2 supports1x402x404x102x2x10A (Top 2 lanes on Module 0; Bottom 2 lanes on Module 1)2x2x10B (Bottom 2 lanes on both modules) Can you confirm first the two XL710-QDA2 are exact the same model? What is the mode you were trying to configure for the XL710-QDA2 that is having the issue? Are both NICs using the same brand and model of cable?Looking forward to your reply.Thanks,Sharon T

[TISHI17]

Hi Sharon,

It's same model.

I used QCU for the first time, but I never setup. It is display only.

It's same brand and same model of cables.

The problematic NIC is manufactured in Dec 2014. The problem-free NIC was manufactured in Jan 2016.

Both manufacturing models possess multiple pieces, and the phenomenon is reproduced.

Best Regards,

Takunii

[idata]

Hi Takunii,

Thank you for the reply. In order to better investigate on this, we need your help to provide below information. Based on the vendor and device ID 8086-1583, both NICs are found to be XL710-QDA2, but with different outcome when connecting to the same switch. We need to further identify the NIC based on the serial number. Please locate the serial number on the white sticker of the physical NIC, you may refer to this link for the serial number format. https://www.intel.com/content/www/us/en/support/articles/000005625/services.html, Click on the Intel Networking Products. Are you connecting the X710 NICs on the same switch port to test? Looking forward to your reply.Thanks,Sharon T

[TISHI17]

Hi Sharon,

I checked the seal of the NIC, but the format seems to be different.

It has 3 lines and is a MAC address, 3digit number and space and 6digits alphanumeric character, part number.

From there i can guess as follows.

3218500084914ADH57401001

9E85680080516ADJ11363001

There are four identical switches, and problems have occurred with all the switches. We changed cable and change connection port, but it was not fixed.

Best Regards,

Takunii

[idata]

Hi Takunii,

Thank you for the information. The last 9 digits or numbers in the serial number form the part number, in your case the first serial number 3218500084914ADH57401001 has the part number of H57401-001 while the second serial number has the part number of J11363-001. The NIC with J11363-001 is the post change version of XL710-QDA2 with part number H574001-001. Please refer to the attached Product Change notification 114233-00 for the changes and issue fixed. The two NICs are the same model but with different revision. Can you clarify which serial number is the working unit? Looking forward to your reply.Thanks,Sharon T

[TISHI17]

Hi Sharon,

The problem is happening in H57401-001.

Best Regards,

Takunii

[idata]

Hi Takunii,

Thank you for the information. The H57401-001 version is the older version that needs revision as stated in Product change notification. Just to double check as I can see there is CRC 0 in the error. Please refer to page 42 in the Technical Specification Update of XL710 about an issue in which " The device may transmit a packet incorrectly, causing the packet to be dropped by the receiver" https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xl710-10-40-controller-spec-update.pdf Please feel free to update me.Regards,Sharon T

[TISHI17]

Hi Sharon,

The counter I sent before is 168 CRC error.

"48. XL710 40 GbE MAC Transmit Packet Issue"

Since it falls under the above, it is necessary to have all NICs exchanged.

Should I contact support team of JAPAN?

Best Regards,

Takunii

[idata]

Hi Takunii,

Thank you for the reply. I sent a PM to you, please check.Regards,Sharon T