INTEL XL710-QDA2 and CRS326-24S+2Q+ link detection problem

[Dimannn]

I have

• INTEL XL710-QDA2
• CRS326-24S+2Q+
• ACD-QSFP-Plus-3m ACD QSFP+, 40G, 3m (DAC)

The problem is that link detection is extremely slow, it takes from a few minutes to an hour. After the link is detected, connection is fine, but if server is rebooted or port reconfigred, connection is lost again for a random period of time.

I tried connecting this DAC directly to 2 ports of switch, link detection is immideate. This also happens if I connect DAC directly to 2 ports netwotk card.

Ethrool result for no link:

root@storage1:/var/www/controller# ethtool ens2f0
Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseCR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 40000baseCR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: no

Ethrool result for good link:

Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseCR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 40000baseCR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: yes

Driver and firmware:

root@storage1:/var/www/controller# ethtool -i ens2f0
driver: i40e
version: 2.14.13
firmware-version: 8.20 0x8000a051 1.2879.0
expansion-rom-version:
bus-info: 0000:65:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[Caguicla_Intel]

Hello Dmitry,

Thank you for the swift response.

We will continue to check this with our engineers. Your patience and kind consideration on this request is highly appreciated. Rest assured that we will get back to you once we heard from them but no later than 2-4 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello Dmitry,

Apologies for the delay on this matter.

Based on our previous conversation, it seems that you only tested one XL710-QDA2. Can you confirm if you have other XL710-QDA2 to test? This is only if you have a spare and no need to purchase a new one.

Hoping to hear from you soon.

We will follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel® Customer Support

[Dimannn]

Hello Crisselle,

We only have one adapter, but yesterday I recieved information from Mikrotik support that they are experiencing the same issue with Mellanox ConnectX®-3 Pro NIC. They are working on resolving the issue.

Regards,

Dmitry,

[Caguicla_Intel]

Hello Dmitry,

Thank you for the update.

There seems to be an issue with compatibility on Mikrotik. We hope that Mikrotik will be able to find the root cause and work on issue resolution. We will also re-escalate this to our engineers and ask for additional suggestion in case there is any.

Let us know if there is further updates from Mikrotik or if you have additional questions. We will get back to you as soon as possible but no later than 2-3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello Dmitry,

Good day!

We hope you don't mind sharing with us if you already get a word from Mikrotik what the issue was or when do you expect to receive an update from Mikrotik. This will indeed help other users experiencing the same issue.

Feel free to let us know if you have additional clarifications or assistance needed on this matter. 

Looking forward to your reply.

We follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello Dmitry,

How are you doing? I hope everything is great.

This is just a follow up for the additional information we requested. We hope you don't mind sharing this with us as this will indeed help other users experiencing the same issue.

Hoping to hear from you soon.

Should there be no response, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[Dimannn]

Hello, Crisselle,

I will post here if I get any update on the issue, this usually takes some time.

Regards,

Dmitry

[Caguicla_Intel]

Hello Dmitry,

Thank you for the reply.

Since Mikrotik already acknowledged the issue and our initial solution to contact the manufacturer, please be informed that we will now proceed closing this request. If you need any additional information in the future, please submit a new question as this thread will no longer be monitored.

Please feel free to post update on this thread as soon as you've heard from Mikrotik so your fellow community users having the same issue will be aware of.

Thank you for your time and cooperation throughout the process. May you have an amazing day ahead!

Best regards,

Crisselle C.

Intel® Customer Support

[MCMan]

FYI, we've been seeing almost the same exact issue with this NIC. Most of the times a link won't establish, sometimes it does, rarely after a good several minutes, and when it does, it's stable.

I can't say if it always eventually links, the most I was able to wait was 15-20 minutes (and no link in that iteration). Re-plugging seemed to "help" more, but it's very sporadic. For a while I thought I had 1 good DAC cable and 2 bad ones, but testing refuted that without further clarity.

It's in a Synology SA3600 NAS running DSM Linux (a fork of Debian). Between maintenance window limitations and equipment location, it'd be quite complicated for us to run a bunch of tests; and I don't have a test environment for 40GbE. 

We are looking into different DAC cables at this point.

[Posted this 8 days ago in a completely random, different thread. The forum system behaved bizarrely.]