Using XL710 with 10Gb link

[MVend1]

Hi!

I am trying to use one of the dual port XL710 adapters to connect with a 10Gb port. For that, I am trying to use an adapter from Mellanox: MAM1Q00A-QSA Cable Module ETH 10GbE 40Gb/s to 10Gb/s QSFP to SFP+. Unfortunately, just plugging it into XL710 and connecting the resulting SFP+ does not lead to an established link. So, I wonder, whether XL710 supports such configuration? My setup:

OS: Ubuntu 14.04

lspci -v:

83:00.0 Ethernet controller: Intel Corporation Ethernet Controller LX710 for 40GbE QSFP+ (rev 01)

Subsystem: Super Micro Computer Inc Device 0000

Physical Slot: 6

ethtool -i eth4

driver: i40e

version: 1.2.37

firmware-version: f4.22.26225 a1.2 n4.24 e12f3

ethtool eth4

Settings for eth4:

Supported ports: [ ]

Supported link modes: 40000baseCR4/Full

40000baseSR4/Full

40000baseLR4/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: 40000baseCR4/Full

40000baseSR4/Full

40000baseLR4/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: Other

PHYAD: 0

Transceiver: external

Auto-negotiation: off

Supports Wake-on: g

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: no

Thank you

[st4]

Hi M_V,

We will check on this.

thanks,

wb

[st4]

Hi M_V,

The Mellanox MAM1Q00A-QSA Cable is not the tested cable. You may refer to the list of supported cable at http://www.intel.ph/content/dam/www/public/us/en/documents/product-briefs/ethernet-xl710-brief.pdf http://www.intel.ph/content/dam/www/public/us/en/documents/product-briefs/ethernet-xl710-brief.pdf

Further just to double check based on your information, are you connecting one port of XL710 to another "network adapter" instead of a switch?

Thanks,

wb

[MVend1]

Hi WB:

True, this adapter is not listed (yet?). In the end, I would need to connect it to Intel 10GbE card (can't tell you exact model this time). However, the link was not established neither with the card nor 10GbE switch.

It seems to me that I miss something very simple, such as autonegotiation of the speed of the link. How can I enable 10000baseXXXX link mode on a port of XL710 card in Linux? As you could see above in ethtool printout, this mode is not supported.

Best,

M_V

[st4]

Hi M_V,

Thanks for the clarification. Let me check on this.

rgds,

wb

[MVend1]

Hi WB:

since it's probably not easy to get your hands on the adapter, let me ask a simpler question:

* does XL710 support 10Gb data rates? It seems that it does support it according to the "Data Brief" of the adapter (

Data rate supported per port: • Optical: 10 GbE/40 GbE, • Direct Attach: 40 GbE).

If it does support 10Gb, how do I enable it in Linux?

Best,

M_V

[st4]

Hi M_V,

You may follow the instruction in the guide below to enable the adapter to configure Port Speed for XL710.

http://www.intel.com/content/dam/www/public/us/en/documents/guides/qsfp-configuration-utility-quick-usage-guide.pdf http://www.intel.com/content/dam/www/public/us/en/documents/guides/qsfp-configuration-utility-quick-usage-guide.pdf

Should see 4 x 10Gb devices with driver loaded

Hope this helps.

rgds,

wb

[MVend1]

Hi WB:

thank you for instructions. However, I do have a problem. Namely, it seems that QCU is not working for us as it should (see below). In addition, in our server, the official name of the card is Supermicro AOC-S40G-i2Q which uses XL710 controller. I wounder if that the cause for the problems.

Here is mu session with QCU:

# ./qcu64e

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

NIC B:D Ven-Dev Mode Adapter Name

=== ===== ========= ====== ====================================================

1) 83:00 8086-1583 N/A Intel(R) Ethernet Converged Network Adapter XL710-Q2

Warning: No adapter selected.

# ./qcu64e /nic=1 /info

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

QSFP+ Configuration modification is not supported by this adapter.

# echo $?

5

# ./qcu64e /exitcodes

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

Exit Codes:

0x0 := Requested action executed successfully.

0x1 := Insufficient privileges to run the utility.

0x2 := Invalid adapter selected.

0x3 := Bad command line parameter.

0x4 := Unsupported link mode selected.

0x5 := Adapter does not support link mode selection.

0x6 := No adapter selected.

0x7 := Memory allocation error.

0x8 := No base driver available.

0x9 := Unsupported base driver version detected.

0xA := QSFP+ configuration for selected adapter not supported by this utility version.

Here are the driver/firmware info:

ethtool -i eth4

driver: i40e

version: 1.2.37

firmware-version: f4.22.26225 a1.2 n4.24 e12f3

Best,

M_V

[DHeki1]

@M_V

For the QCU utility to work correctly, you need to update the firmware and have the latest Linux driver (v1.2.37) loaded. It looks like you have the latest Linux Driver.

https://downloadcenter.intel.com/download/24769/NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series

Be sure to reboot after running NVM Update before running QCU for it to work properly.

My XL710 Card:

driver: i40e

firmware-version: f4.33.31377 a1.2 n4.85 e1930

- David

[MVend1]

Hi David:

strange, I get from NVM Update utility (run in linux) that firmware is up-to-date:

# ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool

NVMUpdate version 1.24.33.08

Copyright (C) 2013 - 2015 Intel Corporation.

WARNING: TO AVOID DAMAGE TO YOUR DEVICE, DO NOT EXIT OR REBOOT OR POWER OFF THE SYSTEM DURING THIS UPDATE

Inventory in progress. Please wait [.......-**]

Num Description Device-Id B:D Adapter Status

=== ====================================== ========= ===== ====================

01) Intel(R) 82599 10 Gigabit Network Conn 8086-1557 03:00 Update not available

02) Intel(R) I350 Gigabit Network Connecti 8086-1521 04:00 Update not available

03) Intel(R) Ethernet Converged Network Ad 8086-1583 131:00 Up to date

Tool execution completed with the following status: All operations completed successfully

ethtool -i eth3

driver: i40e

version: 1.2.37

firmware-version: f4.33.31377 a1.2 n4.42 e18a3

Do I have to unplug the server from mains? Or is the reboot sufficient?

Best,

M_V

[DHeki1]

@M_V,

Looking at your post above:

ethtool -i eth4

driver: i40e

version: 1.2.37

firmware-version: f4.22.26225 a1.2 n4.24 e12f3

And your current post:

ethtool -i eth3

driver: i40e

version: 1.2.37

firmware-version: f4.33.31377 a1.2 n4.42 e18a3

Shows the Firmware to have been updated between each post.

I upgraded the firmware on (8) XL710 cards without powering off, a reboot was sufficient. Same with running the QCU utility.

- David

[MVend1]

I guess I haven't rebooted after the update. However, even now, I still get 'no support' message from QCU. I do wonder if its due to the fact that its Supermicro-branded version?

# ./qcu64e /nic=1 /info

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

QSFP+ Configuration modification is not supported by this adapter.

# echo $?

5

# ethtool -i eth4

driver: i40e

version: 1.2.37

firmware-version: f4.33.31377 a1.2 n4.42 e18a3

bus-info: 0000:83:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

[DHeki1]

What does './qcu64e' show after the reboot? You're looking for something in the Mode to look like (1x40 or 4x10, etc).

Your Card:

NIC B:D Ven-Dev Mode Adapter Name

=== ===== ========= ====== ====================================================

1) 83:00 8086-1583 N/A Intel(R) Ethernet Converged Network Adapter XL710-Q2

My Card:

NIC B:D Ven-Dev Mode Adapter Name

=== ===== ========= ====== ====================================================

1) 03:00 8086-1584 4x10 Intel(R) Ethernet Converged Network Adapter XL710-Q1

Not sure about the Supermicro OEM vs Intel Branded.

- David

[MVend1]

It still shows N/A:

# ./qcu64e

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

NIC B:D Ven-Dev Mode Adapter Name

=== ===== ========= ====== ====================================================

1) 83:00 8086-1583 N/A Intel(R) Ethernet Converged Network Adapter XL710-Q2

I'll try to power-circle it on Monday. Thank you very much for help!

M_V

[MVend1]

Unfortunately, no change after power-circle. QCU and ethtool show the same info:

ethtool -i eth3

driver: i40e

version: 1.2.37

firmware-version: f4.33.31377 a1.2 n4.42 e18a3

./qcu64e /nic=1 /info

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

QSFP+ Configuration modification is not supported by this adapter.

./qcu64e

Intel(R) QSFP+ Configuration Utility

QCU version: v1.24.35.01

Copyright(C) 2015 by Intel Corporation.

Software released under Intel Proprietary License.

NIC B:D Ven-Dev Mode Adapter Name

=== ===== ========= ====== ====================================================

1) 83:00 8086-1583 N/A Intel(R) Ethernet Converged Network Adapter XL710-Q2

Warning: No adapter selected.

Maybe we managed to get an adapter that does not support 4 x 10Gb?

Best

M_V

[st4]

Hi M_V,

Thank you for taking time to further test. let me further check on this.

rgds,

wb

[st4]

Hi M_V,

Can you please check with SuperMicro if there is similar utility to configure the ports?

Thanks,

wb