Problem establishing link with back-to-back XL710-QDA2 NICs connected with a Molex zQSFP+28G DAC @ 40G.

[EKief1]

I am trying to connect two ports different XL710-QDA2 NICs back to back using a Molex zQSFP+28G 30AWG 2 meter DAC.

Details may be found at: https://www.digikey.com/product-detail/en/molex-llc/1002971201/WM11888-ND/5361443

xxx@atlas-1:~$ sudo ethtool p2p1

Settings for p2p1:

Supported ports: [ ]

Supported link modes:  40000baseCR4/Full 

            40000baseSR4/Full 

            40000baseLR4/Full 

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Advertised link modes: 40000baseCR4/Full 

Advertised pause frame use: No

Advertised auto-negotiation: Yes

Speed: Unknown!

Duplex: Unknown! (255)

Port: Other

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x0000000f (15)

    drv probe link timer

Link detected: no

The driver and NVRAM are at the latest and greatest revs. and both systems were powered down and rebooted from the ground up.

Both servers are running Ubuntu server 16.04 LTS.

On one of the servers, I got from dmesg:

xxx@atlas-1:~$ sudo dmesg | grep -i p2p1

[  4.718455] i40e 0000:42:00.0 p2p1: renamed from eth6

[  10.564208] IPv6: ADDRCONF(NETDEV_UP): p2p1: link is not ready

[  99.591988] i40e 0000:42:00.0 p2p1: Cannot read module EEPROM memory. No module connected.

[ 285.778865] i40e 0000:42:00.0 p2p1: Cannot read module EEPROM memory. No module connected.

On the other server, I got:

xxx@atlas-2:~$ sudo dmesg | grep -i p2p1

[  4.515436] i40e 0000:42:00.0 p2p1: renamed from eth6

[  10.528229] IPv6: ADDRCONF(NETDEV_UP): p2p1: link is not ready

This looks like I have to somehow fix the EEPROM on the first server.

I have a couple questions:

1. Do the XL710-QDA2 NICs support the QSFP+28 DACs? According to the DAC specification, they should sense the connector configuration and adapt.
2. Is there some kind of magic tweak that has to be done in the QSFP+ tool? I saw nothing in there that pertains to this issue.
3. How do I fix the EEPROM issue on the first server?

Thank you for your time,

Ed Kiefer

[EKief1]

Quick follow-up:

Shutdown both systems, reseated the QSFP+ connectors, with the results fixing the EEPROM issue, but no connectivity joy:

Server1:

xxx@atlas-1:~$ sudo dmesg | grep -i p2p1

[  4.663459] i40e 0000:42:00.0 p2p1: renamed from eth5

[  10.637673] IPv6: ADDRCONF(NETDEV_UP): p2p1: link is not ready

Server2:

xxx@atlas-2:~$ sudo dmesg | grep -i p2p1

[  4.707076] i40e 0000:42:00.0 p2p1: renamed from eth6

[  10.781448] IPv6: ADDRCONF(NETDEV_UP): p2p1: link is not ready

So back to questions 1&2 above.

Thank you for your time,

Ed Kiefer

[EKief1]

Works now regardless of whether it's officially supported or not.