Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4813 Discussions

INTEL XL710-QDA2 and CRS326-24S+2Q+ link detection problem

Dimannn
Beginner
3,883 Views

I have

  • INTEL XL710-QDA2
  • CRS326-24S+2Q+
  • ACD-QSFP-Plus-3m ACD QSFP+, 40G, 3m (DAC)

The problem is that link detection is extremely slow, it takes from a few minutes to an hour. After the link is detected, connection is fine, but if server is rebooted or port reconfigred, connection is lost again for a random period of time.

I tried connecting this DAC directly to 2 ports of switch, link detection is immideate. This also happens if I connect DAC directly to 2 ports netwotk card.

Ethrool result for no link:

root@storage1:/var/www/controller# ethtool ens2f0
Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseCR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 40000baseCR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: no

Ethrool result for good link:

Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseCR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 40000baseCR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: yes

Driver and firmware:

root@storage1:/var/www/controller# ethtool -i ens2f0
driver: i40e
version: 2.14.13
firmware-version: 8.20 0x8000a051 1.2879.0
expansion-rom-version:
bus-info: 0000:65:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

0 Kudos
29 Replies
Caguicla_Intel
Moderator
2,872 Views

Hello Dimannn,


Thank you for posting in Intel Ethernet Communities. 


Please provide the following information that would help in checking your query.

1.) What is the exact Operating System used?

2.) Please provide photos of the optical modules used focusing on the markings(white sticker). 

3.) Markings of the adapter. You may refer to the link below on where to find the PBA and Serial number. Providing front and back photos of the adapter focusing on the markings (white sticker) found on the physical card will be highly appreciated for us to double check on it.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html


We've also sent an email for you to attach the photos that we are requesting for. Please be reminded not to post the photos on this thread to avoid exposing confidential information.


Awaiting to hear from you. 


Should there be no response from you, I’ll make sure to reach out after 3 business days.


Best regards,

Crisselle C

Intel® Customer Support


0 Kudos
Dimannn
Beginner
2,868 Views

Hello Crisselle,

Thank you for your reply.

OS is Ubuntu 20.04.2 LTS, kernel 5.4.0-66-generic.

Photos of DAC and adapter has been sent to you.

 

I also would like to post dmesg output:

[Tue Mar 2 11:39:17 2021] i40e 0000:19:00.1 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[Tue Mar 2 11:39:17 2021] i40e 0000:19:00.1: Added LAN device PF1 bus=0x19 dev=0x00 func=0x01
[Tue Mar 2 11:39:17 2021] i40e 0000:19:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 16 RSS FD_ATR FD_SB NTUPLE CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[Tue Mar 2 11:39:17 2021] i40e 0000:65:00.0: fw 8.2.64244 api 1.13 nvm 8.20 0x8000a051 1.2879.0
[Tue Mar 2 11:39:17 2021] i40e 0000:65:00.0: MAC address: 40:a6:b7:28:be:d0
[Tue Mar 2 11:39:17 2021] i40e 0000:65:00.0: FW LLDP is enabled
[Tue Mar 2 11:39:17 2021] i40e 0000:65:00.0: PCI-Express: Speed 8.0GT/s Width x8
[Tue Mar 2 11:39:17 2021] i40e 0000:65:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 16 RSS FD_ATR FD_SB NTUPLE CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[Tue Mar 2 11:39:17 2021] i40e 0000:65:00.1: fw 8.2.64244 api 1.13 nvm 8.20 0x8000a051 1.2879.0
[Tue Mar 2 11:39:18 2021] i40e 0000:65:00.1: MAC address: 40:a6:b7:28:be:d1
[Tue Mar 2 11:39:18 2021] i40e 0000:65:00.1: FW LLDP is enabled
[Tue Mar 2 11:39:18 2021] i40e 0000:65:00.1: PCI-Express: Speed 8.0GT/s Width x8
[Tue Mar 2 11:39:18 2021] i40e 0000:65:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 16 RSS FD_ATR FD_SB NTUPLE CloudF DCB VxLAN Geneve NVGRE PTP VEPA

0 Kudos
Caguicla_Intel
Moderator
2,865 Views

Hello Dimannn,


Appreciate your swift response.


Please allow us to further check on this request. We will get back to you as soon as possible but no later than 2-4 business days.


Hoping for your kind patience. 


Best regards,

Crisselle C

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,857 Views

Hello Dimannn,


Good day!


Before we further check this query with our engineers, can you confirm if you have tried using Intel branded QSFP+ modules and cables as mentioned from the link below? Please also share how many adapters are affected by this issue. 

QSFP+ Modules/Cables Compatible with Intel® Ethernet Server Adapter XL710 Series 

Note: Other brands of QSFP+ SR optical modules don't work with the Intel® Ethernet Converged Network Adapters.

https://www.intel.com/content/www/us/en/support/articles/000006051/network-and-i-o/ethernet-products.html


Looking forward to hear from you.


Should there be no response from you, I’ll make sure to reach out after 3 business days.


Best regards,

Crisselle C

Intel® Customer Support


0 Kudos
Dimannn
Beginner
2,830 Views

Hello!

 

I have replaced DAC cable with Intel XLDACBL3 compatible cable and I'm experiencing exectly the same issue.

I only have one adapter.

0 Kudos
Caguicla_Intel
Moderator
2,760 Views

Hello Dimannn,


Thank you for providing the requested information.


Please allow us to investigate this issue with our engineers. We will get back to you as soon as possible but no later than 2-4 business days.


Hoping for your kind understanding.


Best regards,

Crisselle C

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,757 Views

Hello Dimannn,


While we are escalating this request, we hope you don't mind sharing the additional information below that would help with the investigation.

1.) Is there any error message when there is a problem with link detection? 

2.) Have you tried installing the adapter on different system or PCIe slot?

3.) What is the model of the motherboard used?

4.) Please provide the System Support Utility (SSU) log of your system. This will allow us to check your Adapter details and configuration. Please refer to the link below for Help Guide for the Intel® System Support Utility. We will send an email so you can share this privately. Please be reminded not to post the logs on this thread to avoid exposing confidential information.

https://www.intel.com/content/www/us/en/support/articles/000008563/software/software-applications.html


Looking forward to your reply.


Should there be no response from you, I’ll make sure to reach out after 3 business days. 


Best regards,

Crisselle C

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,685 Views

Hello Dimannn,


How are you doing? 


This is just a follow up for the additional information we are requesting for. 

1.) Is there any error message when there is a problem with link detection? 

2.) Have you tried installing the adapter on different system or PCIe slot?

3.) What is the model of the motherboard used?

4.) Please provide the System Support Utility (SSU) log of your system. This will allow us to check your Adapter details and configuration. Please refer to the link below for Help Guide for the Intel® System Support Utility. Please be reminded not to post the logs on this thread to avoid exposing confidential information.

https://www.intel.com/content/www/us/en/support/articles/000008563/software/software-applications.html


Also, we have escalated this request to our higher level and would like to clarify below information.

You mentioned "I tried connecting this DAC directly to 2 ports of switch, link detection is immediate. This also happens if I connect DAC directly to 2 ports network card."

Based on the link, the CRS326-24S+2Q+ is a switch which the XL710-QDA2 should be connected. Can you please clarify when does the issue exactly occur? 

https://mikrotik.com/product/crs326_24s_2q_rm 


Looking forward to your reply.


Should there be no response from you, I’ll make sure to reach out after 3 business days.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,654 Views

Hello Dimannn,


Good day!


This is just a follow up message for the additional information that we are requesting. If you need more time gathering the details, please let us know so we can adjust our follow up date.


Hoping to hear from you soon.


We will follow up after 3 business days if we don't receive your reply.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Dimannn
Beginner
2,644 Views

Hello, Crisselle.

Sorry for late reply. I had troubles running  SSU on Ubuntu 20.04, so i moved the adapter to Windows 10 machine. 

1) There is no error log, except the message that the link in down. After some time link just becomes up without additional logs.

2,3) Adaprer was initially istalled on https://www.supermicro.com/en/products/system/1U/1019/SYS-1019P-WTR.cfm with Ubuntu 20.04. We have moved adapter to Gigabyte H310N 2.0 based PC with Windows 10 and the same behaviour is seen. 

I clarify about direct connection of 2 ports: when I connect mikrotik switch with intel adapter using DAC cable I experience problem with link detection. My first thought was that the problem is caused by cable. Both the switch and the adapter have 2 qsftp+ ports.  So I decided that if the problem is really caused by the cable, then if I connect the cable to 2 ports of the same device, I shall exprerience the same problem as when I connect 2 devices using this cable. But that was not true: if I connect the cable to 2 ports of switch, or if I connect the cable to 2 ports of adapter, everithing works well. This makes me think that the problem is not caused by cable.

I can susspect that the problem is not cable, OS or paltform specific.

SSU data will be sent to support ticket.

I will also try to find a third qsfp+ device to try to find out is the problen actually caused by Microtik switch or Intel adapter, or maybe their mutual misunderstanding.

 

0 Kudos
Caguicla_Intel
Moderator
2,636 Views

Hello Dimannn,


No problem at all. We highly appreciate your effort in extracting SSU logs.


Please allow us to continue investigating the issue with our engineers. Rest assured that we will send an update as soon as possible but no later than 2-3 business days.


Hoping for your kind patience.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,622 Views

Hello Dimannn,


Apologies for the delay on this matter.


We already heard an update from our engineering team. Please refer to details below for their feedback.


Have you tried swapping out the Adapter for another one? Same model or different? This would help rule out a problem with the Hardware vs some configuration or compatibility issue.


Awaiting to hear from you.


Should there be no response from you, I’ll make sure to reach out after 3 business days.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,578 Views

Hello Dimannn,


Good day!


This is just a follow up to check if you have read our previous message. If yes, kindly share the additional information requested for us to further investigate the issue.


Hoping to hear from you soon.


We will follow up after 3 business days in case we don't receive a reply.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,561 Views

Hello Dimannn,


How are you doing?


Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Dimannn
Beginner
2,545 Views

Hello, Crisselle.

 

We are now in the process of purchasing another non-Intel based QSFP card to fulfill your request.

 

Regards,

Dmitry.

0 Kudos
Caguicla_Intel
Moderator
2,538 Views

Hello Dimannn,


Thank you for the update.


There seems to be a misunderstanding regarding our latest update. To provide clarification on this one, we have not require you to purchase a new adapter. If you have a spare card that you can use for testing, it would be better to further isolate the issue. Otherwise, please let us know so we can check other options as purchasing a new adapter requires effort and is not cost effective.


Let us know if you have questions and clarifications.


Looking forward to your reply.


We will follow up after 3 business days in case we don't hear from you.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Dimannn
Beginner
2,511 Views

Hello Crisselle,

 

We have purchased Broadcom OCE14401-UX (EOL) QSFP+ adapter. 

 

Intel to Broadcom - link OK

Broadcom  to Mikrotik - link OK

Intel to Mikrotik - link problem.

0 Kudos
Caguicla_Intel
Moderator
2,506 Views

Hello Dimannn,


Appreciate your effort in sending an update and isolating the issue.


Please allow us to forward your reply to our engineers and check the next step with them. We will get back to you as soon as we heard from them but no later than 2-4 business days. 


Hoping for your kind patience.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Caguicla_Intel
Moderator
2,457 Views

Hello Dimannn,


Thank you for the patience on this matter.


We received an update from our engineers and they have suggested to contact the switch manufacturer (Mikrotik) first to determine if there is a configuration or compatibility issue. Please also ensure that you have the latest firmware and driver for the switch. 


Feel free to let us know if you have additional questions and clarifications.


We look forward to your reply.


Should there be no response, we will follow up after 3 business days.


Best regards,

Crisselle C.

Intel® Customer Support


0 Kudos
Dimannn
Beginner
2,401 Views

Hello Crissele!

I contacted Mikrotik support the same day I started contacting you. Their suggestions, including new beta firmware, unfortunately, did not help either. 

We also noticed that Broadcom card from my previous post works well, so we decided to use it. So there is no rush with the issue and we are able to wait until engineers have some advance on the problem, please notify me when there is any.

Thank you for your help!

Regards,

Dmitry

0 Kudos
Reply