xl710-qda2 to xl710-qda2 without a switch

[Robe30]

I'm having trouble when trying to network a server and pc together using two xl710-qda cards. Both systems are running Opensuse Linux distro.

I know that direct communication between the two will work if the NIC supports auto-MDIX, but I am not able to establish the 40gbe connection from my pc to the server.

Am I wrong to assume that this is possible with two xl710?

[Robe30]

Through searching these forums, it seems that connecting them directly without a switch should be fine. I'm still not seeing any connection though and am not sure how to troubleshoot this.

[Mike_Intel]

Hello Robe30,

Thank you for posting in Intel Ethernet Communities. 

By the way, have you tried connecting them using a switch? Did it worked?

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

No, unfortunately I do not have a switch I can use with these cards. But everything I've read has suggested you don't need a switch.

Both cards are recognized in the network devices of the computers, but I can't seem to get a ping through.

[Mike_Intel]

Hello Robe30,

Please help clarify or provide the following details for us to further check this issue.

1. What is the LAN cable that you are using to connect these 2 systems?
2. Have you tried a different LAN cable?
3. What is the IP addresses that you set for both systems?
4. Can you check if there's an LED lights on the NIC ports of the 2 systems?
5. Please clarify if there is not connection at all since you mentioned you cannot ping the other system or there is a connection but you cannot establish 40GbE connection?

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

Hi Mike,

1. Panduit OM3 30m (fxtrp5n5nanf115)
2. I have two of the OM3 cables mentioned above, I've tried both of them. Unfortunately I don't have any other fibre or copper cables to try
3. I set one to 192.168.10.10 and the other to 192.168.10.11
4. It looks like there are two LEDs for each port on the back of each XL710. I see a red light, and possibly an orange light for the other one (might just be light bleeding from the red LED).
5. There seems to be no connection at all. I connect all of the hardware and configure the static IP for both systems/NICs, but the LED lights on the back of the cards stay red and they can't ping/do anything with each other. This initially made me think it could be my cable, but everything I've read online seems to indicate that 30m OM3 should be fine. Am I wrong?

Additional note: I'm using 2x Intel - QSFP+ transceiver module - 40 Gigabit LAN (E40GQSFPSR)

Thanks

[Mike_Intel]

Hello Robe30,

Thank you for the update.

Upon checking, you are using a 3rd party cable. 3rd party cables won't work with Intel Adapter X710 Series.

For more information and for the list of compatible cables, please open this link below:

https://www.intel.com/content/www/us/en/support/articles/000007045/network-and-i-o/ethernet-products.html

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

This is not a good answer to my question because:

1. Intel does not make QSFP+ optical cables, so the only cables available are 3rd party by definition
2. Nowhere in your link is there a list of compatible fiber cables, nor is there any mention of 40gbase-sr4 or QSFP+

If you have a QSFP+ MPO-MPO fiber-optic cable that is verified compatible with Intel SR4 transceiver modules, please let me know. I have not found any mention of this anywhere.

[Mike_Intel]

Hello Robe30,

I apologize for the link that I provided. Please refer to this link instead for the QSFP+ Modules/Cables Compatible with Intel® Ethernet Server Adapter XL710 Series.

https://www.intel.com/content/www/us/en/support/articles/000006051/network-and-i-o/ethernet-products.html

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

That link does not recommend any QSFP+ fiber cables. It only recommends direct attach copper cables. It does in fact state that the E40GQSFPSR module (which I'm using) is compatable with XL710. So I see no problems based on the link you posted.

I'm willing to believe my cable is the problem, but only if there is a logical explanation for what is wrong with mine.

[Mike_Intel]

Hello Robe30,

I do understand your side since you are speaking technically base on the composition of the cable. However, please also understand that there are also other customers who have issues when using a 3rd party cables and it also states in our website that 3rd party cables don't work with Intel® Ethernet Converged Network Adapters.

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Mike_Intel]

Hello Robe30,

I just want to check if you tried a compatible cable and how was the testing?

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

I have no reason to believe my cable is incompatible, and intel has no list of validated optical cables. We're probably going to switch away from these intel NICs instead of trying to guess which optical cables work.

Thank you for the help

[Mike_Intel]

Hello Robe30,

I am sorry to hear that. That is why we have a public document on our website stating the compatible cables and using other cables will not work. Having said that, do you have other questions or clarifications?

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Mike_Intel]

Hello Robe30,

I just want to check if you still have questions or clarifications regarding this thread.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Mike_Intel]

Hello Robe30,

I have not heard back from you so I will close this inquiry now. 

If you need further assistance, please post a new question. 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Mike_Intel]

Hello Robe30,

By the way, upon checking. Let me share this link for the fiber module that he is using E40GQSFPSR which mentioned for back to back connection, need to configure the same speed for the 2 XL710-QDA2.

https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ethernet-qsfp-optics-brief.pdf

And also, if you can provide the output of ethool -i output for me to check the firmware of your XL710-QDA2.

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

Hi Mike,

I have been using E40GQSFPSR this whole time.

As an update, I got a copper direct attach cable (so not using E40GQSFPSR fiber module) and I can now communicate between the two XL710-QDA2's. The only problem is now that I'm limited to just a few meters of cable (as opposed to 30m of optical fiber).

I was also able to try a couple different mpo-mpo om3 fiber cables that also did not work. So it seems like somehow either the E40GQSFPSR fiber module or optical cables are causing problems? No idea why copper is working but optical isn't.

The output of ethtool -i is:

driver: i40e

version: 2.9.21

firmware-version: 7.00 0x80004c97 1.2154.0

expansion-rom-version:

bus-info: 0000:02:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

I'm not sure how to "configure the speed" of the fiber module (as you say). I'm able to verify using the intel ethernet tool that both XL710-QDA2 are in 2x40 mode.

Am I supposed to configure the fiber module somehow? Maybe this is why copper is working and fiber isn't?

[Mike_Intel]

Hello Robe30,

Thank you for the update. Let me further check this and get back to you.

Please give me 24 to 48 hours to provide an update.

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Mike_Intel]

Hello Robe30,

By the way, what is the brand and model of the cable that you used during this last test?

Are you still using Panduit OM3 30m (fxtrp5n5nanf115)?

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

[Robe30]

HI Mike,

Yes, I'm trying to use Panduit OM3 30m (fxtrp5n5nanf115). In addition, I also tried the following two optical cables (which did not work):

Molex 1062250001 (10m)

Finisar Corporation FCCN410QD3C10 (this one has its own fiber module, so I didn't expect this one to work anyway)

Thanks