i40e x710 4x10G card not working anymore with DAC

[JGuy]

Using an updated driver, I cannot seem to get my interfaces up. Looks like the driver is not allowing the NIC to work due to the SFP+ module type, which worked previously on my Debian Stretch host. Clearly the NIC is detecting and reading the DAC, but I don't understand why it is not working. This is a 40G-breakout cable with 40-to-4x10G SFP+ DAC (TwinAx). I have tried adding the following to work around it, but it still does not work:

root@lab4:~# cat /etc/modprobe.d/i40e.conf

options i40e allow_unsupported_sfp=1

This was also discussed here, but there was no followup:

https://sourceforge.net/p/e1000/mailman/message/36029185/ Intel Ethernet Drivers and Utilities / Mailing Lists

Output from dmesg:

[ 4.439190] i40e 0000:08:00.3: PCI-Express: Speed 8.0GT/s Width x8

[ 4.447918] i40e 0000:08:00.3: Features: PF-id[3] VFs: 32 VSIs: 34 QP: 8 RSS FD_ATR FD_SB NTUPLE CloudF DCB VxLAN Geneve NVGRE PTP VEPA

[ 4.453414] i40e 0000:08:00.2: Rx/Tx is disabled on this device because an unsupported SFP+ module type was detected.

[ 4.465288] i40e 0000:08:00.2: Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.

Other pertinent output:

root@lab4:~# ethtool -m eth6

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 3m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400003

Vendor rev : A

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF13470035GDT

Date code : 131122

root@lab4:~# ethtool -i eth6

driver: i40e

version: 2.4.3

firmware-version: 6.01 0x800034af 1.1747.0

expansion-rom-version:

bus-info: 0000:08:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

root@lab4:~# ethtool eth6

Settings for eth6:

Supported ports: [ ]

Supported link modes: Not reported

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: Not reported

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: Other

PHYAD: 0

<span ...

[idata]

Hi Ninwardspiral,

Thank you for posting in Wired Communities. Can you share what is the exact brand and model of the 40G break out cable used? Thanks,Sharon T

[JGuy]

Hi Sharon,

This is a generic Amphenol 3 meter breakout 40G QSFP+ to 4x10G SFP+ DAC.

All of the generic Amphenol straight 10G SFP+ DACs appear to work fine.

I know these used to work prior to updating the firmware and drivers...

[idata]

Hi Ninwardspiral,

Thank you for the information. I would like to clarify further information in order to better assist you.1) Based on your NIC descreption, are you using an X710-DA4? If this is the model, it is recommended to use the cable listed in the website below: https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html2)The breakout cable are applicable for use on XL710 series (not X710 series), and we recommended to use cable model listed in the link below: https://www.intel.com/content/www/us/en/support/articles/000006051/network-and-i-o/ethernet-products.html Having said that, can you help clarify :1) Are you saying the Amphenol 3 meter break out 40Gb QSFP+to 4x10GSFP+DAC used to work on this X710-DA4 prior you upgraded to firmware 6.01?2) Are the generic Amphenol straight 10 SFP+ (which is another cable) you refer here works on the X710-DA4 ? Looking forward to your reply.Thanks,Sharon T

[JGuy]

Hi Sharon,

Yes, this is an X710-DA4. Let me clarify the connections made using the breakout cable.

On the network switch is a 40G QSFP+ port, which has been configured as 4x10G. So the QSFP+ is inserted in the switch. Then the SFP+ connections are in the Intel NIC.

I am certain the EEPROM of the SFP+ module is correct, and works in other NICs and devices.

1) Yes, these generic DACs worked fine in FW 5.01, with the 1.4, and 2.0 driver versions. I did try downgrading the driver to 2.0.33 (leaving the FW at 6.01), but the breakout SFP+ still does not work.

2) Yes, the Amphenol straight SFP+ to SFP+ 10G cable works fine. This module is almost identical to the one used in the breakout, though it has a few more transceiver codes set in the EEPROM as expected.

3) Just to try some more tests, I tried an identical Dell branded 40G-to-4x10G breakout cable, because it is just an amphenol cable with a different sticker and part number. The Dell breakout actually works!

I have provided the EEPROM output from both, so you can see there is very little difference (see below).

Is there a documented list of all Passive DACs that are whitelisted for use in the X710 cards?

It would be good to understand why the NIC is blocking the Amphenol, and allowing the Dell.

Thanks for your help looking at this.

Cheers,

Jason

# Amphenol (Generic from Cables on demand) SFP+ side of a 40-to-4x10 breakout

root@lab4:~# ethtool -m eth7

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 5m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400005

Vendor rev : A

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF13490056MGJ

Date code : 131205

# Dell (Amphenol) SFP+ side of a 40-to-4x10 breakout

root@lab4:~# ethtool -m eth8

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

<span style="c...

[idata]

Hi Ninwardspiral,

Thank you for the clarification and additional information. Let me further investigate on this. By the way, just to double check have you sent the output of "ethtool -i

Regards,

Sharon T

[idata]

Hi Ninwardspiral,

Please refer to below information:1) If the cable stopped working after the firmware update, you might want to update the firmware again in case there is issue during the firmware update process.You can try downgrading back to 5.x FW and use applicable drivers. Test the cable again to confirm it works using that firmware. Then try update to firmware 6.01 again. 2) Please check if your X710-DA4 is a retail unit. You can check the Vendor and Device ID based on the information here. https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-i-o/ethernet-products.html You should see the VendorID:8086 and DvID:1572 for the retail version.3) Can you try a different cable length to see if this will make any difference? 4) With regards to the Dell* cable, it is possible there is difference other than labeling between the generic and the Dell* version of the Amphenol which we do not have the information.Please feel free to update me.Thank you.Regards,Sharon T

[idata]

Hi Ninwardspiral,

Please feel free to update me.Thanks,Sharon T

[JGuy]

Hi Sharon,

1) Unfortunately at this point I cannot downgrade the firmware since there were many fixes we needed in the latest FW and driver. We recently had to upgrade to do some 802.1ad validation. So I hope there is a way to debug this and identify specifically what is not allowing the module to be used.

2) I have several of these cards, and they are the retail version:

root@lab2:~# lspci -nn | grep -i 'X710'

08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

08:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

08:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

08:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

09:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

09:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

09:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

09:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

3) I have tried 2m, 3M and 5M, with the last one the one I realized matches the Dell DAC.

Can you tell me if there is a knob in the driver that can be used to ignore some embedded check of the DAC part number, vendor, etc?

Perhaps something that can be passed to the driver via modprobe (something like ignore_whitelist=1). I think the ixgbe driver has an allow_unsupported option...is there anything similar for i40e?

4) From a layer 1 point of view, all of the electrically pertinent details of the DAC modules are identical between the vendors (dell and generic). If there is simply a whitelist built into the firmware, or driver, I would be interested in understanding how to add to that list.

Thanks,

Jason

[idata]

Hi Ninwardspiral,

Thank you for the information. I will have to check on your inquiries.Regards,Sharon T

[idata]

Hi Ninwardspiral,

There is no whitelist. Can you the part number of the X710-DA4 you have. This can be found on the white label of the physical network card which is near 2D barcode. The format goes like 15 alphanumeric + 6 alphanumeric + 5 alphanumeric - 3 digits.Thanks.Regards,Sharon T

[JGuy]

Hi Sharon,

This is the numbers from the card/sticker I can see:

6805CA302920008 4414AD H58361 002

Thanks,

Jason

[idata]

Hi Ninwardspiral,

Thank you for the information. I will check on this.Regards,Sharon T

[idata]

Hi Ninwardspiral,

Just to double check, can you try connect using a regular SFP-SFP cable? Even try back to back connection. Please feel free to update me.Thanks,Sharon T

[idata]

Hi Ninwardspiral,

Please feel free to update me the result of using regular SFP-SFP cable and back to back connection.Thanks,Sharon T

[JGuy]

Hi Sharon,

I did validate that an SFP+ to SFP+ of the same brand works fine back to back, and to other devices. The interesting thing here is the SFP+ module is basically the same as the ones used on the 10G SFP+ side of the 40G-4x10G breakout cables. Apparently cable length does not matter because the results are always the same regardless of length.

Thanks,

Jason

[idata]

Hi Ninwardspiral,

Thank you for the update. I will check if there is any information to share. Regards,Sharon T

[idata]

Hi Ninwardspiral,

I sent a PM to you, please reply me from there.Thanks,Sharon T

[JGuy]

Hi Sharon,

Interestingly, we have had a bunch of customer-reported issues similar to this one. Most of them are more like Edgar's issue, where the Intel 10G-SR optics are not working properly with the Cumulus Linux switches.

At this point, it may be best to open this as a support case with Intel.

I have been trying to downgrade one of my cards to see if the problem is in the new firmware. Sadly, the NVM downgrade reports an error, but appears to have changed it. Perhaps a bug?

root@lab4:~/Intel_NIC/nvm/XL710_downgrade_6.0-to-5.05/Linux_x64# ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool

NVMUpdate version 1.30.22.1

Copyright (C) 2013 - 2017 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

Inventory in progress. Please wait [.-********]

Num Description Ver. DevId S:B Status

=== ======================================== ===== ===== ====== ===============

01) Intel(R) Ethernet Converged Network 6.01 1572 00:008 Update

Adapter X710-4 available

02) Intel(R) Ethernet Converged Network 6.01 1572 00:009 Update <<<<<<<<<<<< Started at 6.01</strong>

Adapter X710-4 available

03) Intel(R) Ethernet Converged Network 6.01 1584 00:065 Update

Adapter XL710-Q1 available

04) Intel(R) 82576 Gigabit Dual Port Network 10C9 00:068 Access error

Connection

05) Intel(R) 82576 Gigabit Dual Port Network 10C9 00:069 Access error

Connection

Options: Adapter Index List (comma-separated), [A]ll, e[X]it

Enter selection:2

Would you like to back up the NVM images? [Y]es/[N]o: y

Update in progress. This operation may take several minutes.

[**-.......]

Reboot is required to complete the update process.

Tool execution completed with the following status: An error occurred accessing the device

Press any key to exit.

root@lab4:~/Intel_NIC/nvm/XL710_downgrade_6.0-to-5.05/Linux_x64# ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool

NVMUpdate version 1.30.22.1

Copyright (C) 2013 - 2017 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

Inventory in progress. Please wait [******+...]

Num Description Ver. DevId S:B Status

=== ======================================== ===== ===== ====== ===============

01) Intel(R) Ethernet Converged Network 6.01 1572 00:008 Update

Adapter X710-4 available

02) Intel(R) Ethernet Converged Network 5.05 1572 00:009 Up to date <<<<<<<<<<<<< Appears to have downgraded this, I will reboot to verify!</strong>

Adapter X710-4

03) Intel(R) Ethernet Converged Network 6.01 1584 00:065 Update

Adapter XL710-Q1 available

04) Intel(R) 82576 Gigabit Dual Port Network 10C9 00:068 Access error

Connection

05) Intel(R) 82576 Gigabit Dual Port Network 10C9 00:069 Access error

Connection

Options: Adapter Index List (comma-separated), [A]ll, e[X]it

Enter selection:x

Tool execution completed with the following status: An error occurred accessing the device

Press any key to exit.

root@lab4:~/Intel_NIC/nvm/XL710_downgrade_6.0-to-5.05/Linux_x64# reboot

root@lab4:~# ethtool -i eth10

driver: i40e

version: 1.6.42

firmware-version: 5.05 0x8000289d 1.1568.0 <<<<<<<<<<<<<<<<<< Looks like it worked.</strong>

expansion-rom-version:

bus-info: 0000:09:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

Now BOTH the generic Amphenol AND the Dell breakout cable are working. Looks like there is a bug in the latest firmware and/or driver.

Unfortunately the ethtool -m output does not work with this driver... I suppose I can prove it by showing the LLDP output from these ports and the switch.

But you can take my word for it that the Dell is in eth10, and the Generic is in eth11.

Here are the ethtool outputs:

Dell Breakout DAC:

----------------------

Settings for eth10:

Supported ports: [ FIBRE ] <<<<<<<<<<<<<<< This was not correct in the new FW and version!</strong>

Supported link modes: 10000baseT/Full <<<<<<<<<<<<<<< This was not correct in the new FW and version!</strong>

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: Not reported

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: external

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: yes

root@lab4:~# ethtool -i eth10

driver: i40e

version: 1.6.42

firmware-version: 5.05 0x8000289d 1.1568.0

expansion-rom-version:

bus-info: 0000:09:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

root@lab4:~# ethtool -m eth10

Cannot get module EEPROM information: Operation not supported

Generic Breakout DAC:

---------------------------

root@lab4:~# ethtool eth11

Settings for eth11:

Supported ports: [ FIBRE ] <<<<<<<<<<<<<<< This was not correct in the new FW and version!</strong>

Supported link modes: 10000baseT/Full <<<<<<<<<<<<<<< This was not correct in the new FW and version!</strong>

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: Not reported

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: external

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: yes<...

[idata]

Hi Ninwardspiral,

Thank you for the additional information provided. Interesting to know the test you did and Amphenol cable works after downgraded to firmware 5.05. Just to double check one thing here. Are you referring this issue happened with Cumulus Linux switches only ? How about other switch model? We will further investigate on this and update this thread accordingly. Thanks,Sharon T

[JGuy]

Hi Sharon,

Yes, I have tested with Cumulus Linux on both Broadcom and Mellanox switches, as well as Cisco switches.

I have not tested to other hosts because I find connecting back-to-back with Intel NICs always seems to work (I assume that is how you folks test).

More information:

I decided to try this with the latest driver, while keeping the firmware on version 5.05.

root@lab4:~# ethtool -i eth10

driver: i40e

version: 2.4.6

firmware-version: 5.05 0x8000289d 1.1568.0

expansion-rom-version:

bus-info: 0000:09:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

root@lab4:~# ethtool eth10

Settings for eth10:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full <<<< Still not right.... it SHOULD actually list many, including "10000baseCR/Full"</strong>

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: 10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: yes

root@lab4:~# ip link show eth10

12: eth10: mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 68:05:ca:2f:7d:c8 brd ff:ff:ff:ff:ff:ff

root@lab4:~# ethtool -m eth10

Cannot get module EEPROM information: Invalid argument

^^^^^ I suppose this is a firmware thing, rather than a driver thing.

root@lab4:~# ethtool -i eth11

driver: i40e

version: 2.4.6

firmware-version: 5.05 0x8000289d 1.1568.0

expansion-rom-version:

bus-info: 0000:09:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

root@lab4:~# ip link show eth11

13: eth11: mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 68:05:ca:2f:7d:c9 brd ff:ff:ff:ff:ff:ff

Here is the output from the Cumulus Linux on a Mellanox 2740 switch, for the DAC initially reported in this post (Generic Amphenol 4x10G_to_40G breakout):

root@cumulus:~# ethtool swp13

Settings for swp13:

Supported ports: [ TP FIBRE ]

Supported link modes: 1000baseT/Full

10000baseT/Full

40000baseCR4/Full

40000baseSR4/Full

40000baseLR4/Full

25000baseCR/Full

50000baseCR2/Full

100000baseSR4/Full

100000baseCR4/Full

100000baseLR4_ER4/Full

10000baseCR/Full

10000baseSR/Full

10000baseLR/Full

10000baseLRM/Full

10000baseER/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Supported FEC modes: None BaseR RS

Advertised link modes: 1000baseT/Full

10000baseT/Full

40000baseCR4/Full

40000baseSR4/Full

40000baseLR4/Full

25000baseCR/Full

50000baseCR2/Full

100000baseSR4/Full

100000baseCR4/Full

100000baseLR4_ER4/Full

10000baseCR/Full

10000baseSR/Full

10000baseLR/Full

10000baseLRM/Full

10000baseER/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: Yes

Advertised FEC modes: Not reported

Speed: 10000Mb/s

Duplex: Full

Port: Other

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Current message level: 0xffffffa1 (-95)

drv ifup tx_err tx_queued intr tx_done rx_status pktdata hw wol 0xffff8000

Link detected: yes

root@cumulus:~# ethtool -m swp13

Identifier : 0x0d (QSFP+)

Extended identifier : 0x00

Extended identifier description : 1.5W max. Power consumption

Extended identifier description : No CDR in TX, No CDR in RX

Extended identifier description : High Power Class (> 3.5 W) not enabled

Connector : 0x23 (No separable connector)

Transceiver codes : 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00

Transceiver type : 40G Ethernet: 40G Base-CR4

Encoding : 0x00 (unspecified)

BR, Nominal : 10300Mbps

Rate identifier : 0x00

Length (SMF,km) : 0km

Length (OM3 50um) : 0m

Length (OM2 50um) : 0m

Length (OM1 62.5um) : 0m

Length (Copper or Active cable) : 3m

Transmitter technology : 0xa0 (Copper cable unequalized)

Attenuation at 2.5GHz : 8db

Attenuation at 5.0GHz : 12db

Attenuation at 7.0GHz : 0db

Attenuation at 12.9GHz : 0db

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400003

Vendor rev : A

Vendor SN : APF134600356WM

Revision Compliance : Revision not specified

Module temperature : 0.00 degrees C / 32.00 degrees F

Module voltage : 0.0000 V

root@cumulus:~# lldpctl swp13

-------------------------------------------------------------------------------

LLDP neighbors:

-------------------------------------------------------------------------------

Interface: swp13, via: LLDP, RID: 28, Time: 0 day, 00:41:02

Chassis:

ChassisID: mac b0:83:fe:c0:4e:f9

SysName: lab4.rdu.cumulusnetworks.com

SysDescr: Debian GNU/Linux 9 (stretch) Linux 4.9.0-3-amd64 # 1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64

MgmtIP: 192.168.121.1

MgmtIP: fe80::5054:ff:fed8:baab

...