x710 4x10G card not working Amphenol cable

[ecost5]

Hi Sharon and Ninwardspiral,

We are having similar problems with the exact same NIC, and by coincidence with the same breakout cable (although we tested many).

Driver and firmware are updated to the latest versions available (2.4.3 and 6.01 respectively):

version: 2.4.3

firmware-version: 6.01 0x800034af 1.1853.0

First cable that does not work: ( QSFP to 4x10G SFP+, amphenol, 2m)

From the manufacturer website: Each QSFP-SFP+ splitter cable features a single QSFP+ connector (SFF-8436) rated for 40-Gbps on one end and (4) SFP+ connectors (SFF-8431), each rated for 10-Gbps, on the other.

https://www.digikey.ch/product-detail/en/amphenol-commercial-products/SF-QSFP4SFPPS-002/SF-QSFP4SFPPS-002-ND/6873743?keywords=SF-QSFP4SFPPS-002%09&cur=CHF&lang=en SF-QSFP4SFPPS-002 Amphenol Commercial Products | Cable Assemblies | DigiKey (exact cable we bought)

ethtool -m output:

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 2m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400002

Vendor rev : A

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF1718002044N

Date code : 170513

ethtool output (after bringing interfaces up):

Supported ports: [ ]

Supported link modes: Not reported

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: Not reported

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: Other

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: no

Second cable that does not work: ( QSFP to 4x10G SFP+, generic brand):

https://www.fs.com/products/21409.html

2m(6.6ft) 40G QSFP+ to 4SFP+ Passive Breakout DAC Cable | FS.COM

ethtool -m output:

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

<p style="colo...

[idata]

Hi Cedgar,

Thank you for posting in Wired Communities. I am sorry to hear what happened. Please refer to information below hopefully can clarify the concern:1) Just want to confirm if the NIC model is a X710-DA4?2) The first two cables you tried are breakout cables, which means one end is a single connection and the other end are with 4 connections. This kind of cable is applicable for XL710 series (XL710-QDA1 or XL710-QDA2 NIC) which has a single 40GB port or dual port 40GB per port. e.g by connecting the breakout cable to a XL710-QDA1, the single 40GB port is subdivided to 10Gbps per port on each port, which means 40Gb is divided to 4 x 10 Gbps. Please refer to our QSFP+ breakout cable product brief, as you can see breakout cable is for the XL710 series NIC https://www.intel.com/content/www/us/en/ethernet-products/optics-cables/ethernet-qsfp-optics-brief.html?_ga=2.34173224.84908822.1519284483-1145062571.1519008255 The third cable you tested works on the X710-DA4 since it is not a breakout cable.3) The cable supported by X710-DA4 is SFP+ DA twin Axial cable up to 10m https://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA44) One of the difference of X710 series and XXV710-DA2 lies in the speed supported. XXV710-DA2 can support up to 25Gbps. This does not determine this will work or work better with the cables you mentioned.Thanks,Sharon T

[ecost5]

Hi Sharon T,

1) Yes we have 2 x710-DA4 that we are trying to connect to switches with QSFP+ ports.

2) Why is it only suitable for XL710 series? I connect the QSFP side of the cable to the switch and the 4x10G connections to the NIC, what is the problem on doing that? In this thread:

which i was referring to in my first message (I actually posted that there), Ninwardspiral is asking you the same exact thing, we have the same problem. And at some point he says that he found a cable that is exactly the same than the amphenol, but its labeled as a Dell cable and it works.

3) So you are telling me that QSFP+ to 4xSFP+ does not work? I am connecting the SFP+ side of the cable to the NIC....

4) I know that that NIC is a 2x25G port NIC. So if it does not work there, how can we even use BREAKOUT cables?

Edgar

[JGuy]

Hi Edgar,

1) Yes we have 2 x710-DA4 that we are trying to connect to switches with QSFP+ ports.

...

4) I know that that NIC is a 2x25G port NIC. So if it does not work there, how can we even use BREAKOUT cables?

I thought the x710-4DA was only a 4x10G NIC... Can you provide the 'ethtool -m' output from the host? What type of switch is it?

Regardless, one thing to consider with a 25G port is the pluggable has to be an SFP28.

If you are connecting the SFP+ to it, make sure auto-negotiation is on for both ends of the link to establish the link in 10G. I have seen native 25G ports not come up with common SFP+ modules. It all depends on vendor.

Cheers,

Jason

[ecost5]

Hi Jason,

First I would like to to clarify something. When I said 2 x710-DA4 I meant that we have two NICs in our set up, and of course they are 4x10G as you thought. -

I provided the 'ethtool -m' output in my first post. But here it goes again:

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 2m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400002

Vendor rev : A

We have a switch that is a bit special (barefoot tofino). It supports QSFP+ and QSFP28, in this case my module is QSFP+. If we don't find a solution with the current NICs, we will return them and go for the 25G and QSFP28 to 4xSFP28, so I guess that should work right?

PS: Did you ever managed to make a QSFP+ to 4xSFP+ breakout cable work with this NIC? As far as I understood you say that before upgrading drivers/firmware your amphenol cable was working, right?

Kind regards,

Edgar

[JGuy]

Hi Edgar,

I have tested the Mellanox 100G switch ports, using the Dell and Generic Amphenol cables, to the Intel X710 SFP+ port. The results are the same as the native 40G switch reported originally in the other thread. There is no reason these should not BOTH work, as they are essentially the same cable.

Before we got the Dell cables, we only had the generic amphenol DACs, and these certainly worked with the older firmware and the 1.4 driver. I can try downgrading the firmware on one card...not entirely sure how to do that.

I would also recommend staying away from FS.com cables, as we have noticed plenty of issues with those in customer networks.

Dell Breakout DAC (WORKING):

root@lab4:~# ethtool eth10

Settings for eth10:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: 10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: yes

root@lab4:~# ethtool -m eth10

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 5m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 601110005

Vendor rev : F

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF12190054F1A

Date code : 120511

Amphenol DAC (NOT WORKING):

root@lab4:~# ethtool eth10

Settings for eth10:

Supported ports: [ ]

Supported link modes: Not reported

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: Not reported

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: Other

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: no

root@lab4:~# ethtool -m eth10

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 3m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400003

Vendor rev : A

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF134600356WM

Date code : 131122

[ecost5]

Thanks for the answer Jason,

Next week we will receive a new breakout cable, if that one does not work either, we will change the NICs. I feel like we are playing the lottery...

Do you recommend any good multi-port NIC? It can be SFP+ or SFP28

Kind regards,

Edgar

[JGuy]

Hi Edgar,

Just to answer your question about what other NIC I would recommend, I thought I would try this Generic Amphenol breakout DAC on a Mellanox Connect-X4 NIC, and it works perfectly! You can see in the output below, that the serial number in the EEPROM output is the same module I previously had in the Intel x710 port, which is depicted under "Amphenol DAC (NOT WORKING)".

So I guess if you want to swap NICs, this is an alternative, and it is a 10G/25G capable NIC.

Cheers,

Jason

root@lab1:~# lspci -s 01:00

01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

root@lab1:~# ethtool eth7

Settings for eth7:

Supported ports: [ Backplane ]

Supported link modes: 1000baseKX/Full

10000baseKR/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Advertised link modes: 1000baseKX/Full

10000baseKR/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: Yes

Link partner advertised link modes: Not reported

Link partner advertised pause frame use: No

Link partner advertised auto-negotiation: Yes

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000004 (4)

link

Link detected: yes

root@lab1:~# ethtool -m eth7

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 3m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400003

Vendor rev : A

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF134600356WM

Date code : 131122

Optical diagnostics support : No

[ecost5]

Thanks Jason.

Today I got an "intel compatible" cable and it does not work... I will return the NICs and get some 25G NIC.

Since the intel cable did not work either, I was thinking that it might be the switches fault, but I do not know how to check which side of the link is the problematic one. Do you know if there is a way other than using ethtool which only

reports something different when both sides of the connection establish a link (even if its down). Is there a way I could debug that? Note that we dont have any device with QSFP ports apart from this one switch i told you before.

[JGuy]

Hi Cedgar,

If you still have the X710 cards, try downgrading the firmware to 5.05 using the NVM downgrade utility. I bet things will start working.

[ecost5]

That would have been good to know, I returned the cables 1 day before you posted that... a bit unlucky..

I made the last cable we ordered work, by disabling auto-negociation in the switch, but only with a generic intel compatible breakout, the amphenol never worked .. ( https://www.fs.com/products/36276.html https://www.fs.com/products/36276.html )

By the way, dont you lose features by downgrading the firmware?

Thanks for eveything!

[JGuy]

Hi Cedgar,

Yes, there are a lot of features lost if you downgrade, making it not optimal. It is a good way to prove that there is something broken between 5.05 and 6.01.

We noticed a similar problem while testing with Intel optics, which are supposed to work. We found there MUST be a problem in the auto-negotiation code, because as soon as we executed 'ethtool -s eth7 advertise 0x80000000000', the links came up. So clearly constraining the advertised link modes to the highest, removes any decision from the auto-negotiation code. I also disabled auto-neg on the switch (Broadcom Trident2+) side of the link, however I am not sure this is necessary once I pinned the host (X710) to only try 10G-SR/Full.

This specific solution is not working with the DACs, but perhaps there is an ethtool option that can force the interface to recognise the 10G side of a breakout DAC. The funny thing is, a 10G to 10G Amphenol DAC cable (essentially using the same electrical module) works fine without changing anything on the switch for Layer1. Furthermore, I do not see any difference in the information in the eeprom. In fact, the EEPROM output is identical.... See supporting output below...

Regardless, I am quite certain the root cause of the problems in the 6.x firmware, is most likely related to:

1) auto-negotiation

2) module detection is not recognising the modules and setting the correct parameters

3) whitelist in firmware (we know this is true for optical modules) and explains this output in dmesg:

[427334.492215] i40e 0000:08:00.2: Rx/Tx is disabled on this device because an unsupported SFP+ module type was detected.

[427334.504168] i40e 0000:08:00.2: Refer to the Intel(R) Ethernet Adapters and Devices User Guide for a list of supported modules.

[427675.493990] i40e 0000:08:00.2 eth8: Autoneg not supported on this phy

Clearly Intel has not done a lot of interop testing here. These problems are really easy to detect, so I hope Sharon opens a bug to get this firmware fixed.

Cheers,

Jason

Amphenol Breakout DAC 10G SFP+ side

----------------------------------------------------------

root@lab4:~# ethtool eth8

Settings for eth8:

Supported ports: [ ]

Supported link modes: Not reported

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: Not reported

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: Other

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: no

root@lab4:~# ethtool -m eth8

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 3m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 78:a7:14

Vendor PN : 624400003

Vendor rev : A

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF134600356WM

Date code : 131122

Amphenol 10G SFP+ DAC

----------------------------------------------------------

root@lab4:~# ethtool eth9

Settings for eth9:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Advertised link modes: 10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

drv probe link timer

Link detected: yes

root@lab4:~# ethtool -m eth9

Identifier : 0x03 (SFP)

Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)

Connector : 0x21 (Copper pigtail)

Transceiver codes : 0x00 0x00 0x00 0x00 0x00 0x04 0x00 0x00

Transceiver type : FC: Copper Passive

Encoding : 0x00 (unspecified)

BR, Nominal : 10300MBd

Rate identifier : 0x00 (unspecified)

Length (SMF,km) : 0km

Length (SMF) : 0m

Length (50um) : 0m

Length (62.5um) : 0m

Length (Copper) : 2m

Length (OM3) : 0m

Passive Cu cmplnce. : 0x01 (SFF-8431 appendix E) [SFF-8472 rev10.4 only]

Vendor name : Amphenol

Vendor OUI : 41:50:48

Vendor PN : 599700003

Vendor rev : C

Option values : 0x00 0x00

BR margin, max : 0%

BR margin, min : 0%

Vendor SN : APF12280036UDG

Date code : 120801

ethtool -e eth8 > eth8_eeprom

ethtool -e eth9 > eth9_eeprom

root@lab4:~# diff eth8_eeprom eth9_eeprom

root@lab4:~#

The first page of the EEPROM is the critical one. These may have different part numbers, but the electrical settings are the same. This is why I suspect there may be a whitelist in the firmware... Not cool.

root@lab4:~# more eth8_eeprom

Offset Values

------ ------

0x0000: 49 02 04 00 05 00 28 80 24 80 2e 80 ff 7f 80 7d

0x0010: b6 0b ...

[idata]

Hi Cedgar,

With regards to the firmware changes, you may refer to page 10 and 15 of below XL710 controller matrix for the changes from firmware 5.05 to 6.01, there are some enhancement such as SR-IOV (Intel® Ethernet Adaptive Virtual Function) and support of 1GbE SX\LX Optical modules. https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-ethernet-controller-feature-matrix.pdfThanks,Sharon T

[VGuir]

Hi Everyone,

After reading the following post, I thought I need to share my experience with the technology.

We have 3 x Lenovo HX nodes with Intel X722 network card.

Using a Mellanox SX1012x switch to interconnect.

We have tried using 2 types of cables now and waiting for a 3rd type.

We have tried using QSFP to SFP+ DAC and SFP+ to SFP+ DAC with QSFP converter. None of which can make the ports on the NIC come up.

The cables are all from Mellanox.

We have spoken to all 3 suppliers. Lenovo, Mellanox and Nutanix and none of them are providing any solutions. The only thing we are told is that the Switch and Nic might be incompatible, but no one knows to why ????

The only way I can get the ports to come up on the NIC or Mellanox switch is to use the SFP+ to SFP+ cable and loop the ports together. Magically they come up and identify the transceivers and the cable as they should.

We have tried turning off or on the Autoneg on both ends. We have forced the QSFP side to 10GB on the switch.

We are now waiting for a breakout cable to arrive as our third option. QSFP to 4 x SFP+ from Mellanox.

Will keep you posted.

[LFerr9]

Hi Vladimir, can I know if you finally solved your problem and how? We have same environnement with Lenovo HX Nutanix, X722 and Mellanox switch. We got breakout cables from Mellanox and are not able to get all ports up. Only few are up and it's not always the same... Thank you.

[JGuy]

Lionel and Vladimir,

Leave the autoneg setting alone, and try adjusting the speeds advertised by the Intel NIC. In linux, you can change the speeds advertised using ethtool. I found the links are more likely to come up if I constrain the advertised speeds to remove the 1G setting.

ethtool -s eth7 advertise 0x80000000000

On the x722, it can do 25G or 10G, so you would need to look at the man page to see what hex value sets the right bits int he advertisement.

HTH,

Jason

[idata]

Hi Cedgar,

Please let us know if you have other assistance needed from Wired Ethernet Community.Regards,Vince T.Intel Customer SupportAgent under contract to Intel