Intel XL710 40G + QSFP+ AOC/DAC-cables - very high latency

[cubewerk]

Hi,

We have 3 high-performance cluster-nodes directly connected via broadcast bond (Proxmox Linux, Kernel 5.15.35-2)

Server 1 -> Server 2

Server 2 -> Server 3

The directly connected setup works flawlessly on other clusters with 10G SFP+. The new QSFP+ 40G direct links however have a very high latency. Bond setup is simple:

iface bond0

inet static address 172.16.0.5/24

5.255.0 bond-slaves enp132s0f2 enp132s0f3

bond-mode broadcast

bond-miimon 100

Test-Results:

10G SFP+:

84:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
84:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

PING 172.16.2.103 (172.16.2.103) 56(84) bytes of data.

64 bytes from 172.16.2.103: icmp_seq=1 ttl=64 time=0.049 ms

64 bytes from 172.16.2.103: icmp_seq=2 ttl=64 time=0.056 ms

64 bytes from 172.16.2.103: icmp_seq=3 ttl=64 time=0.056 ms

64 bytes from 172.16.2.103: icmp_seq=4 ttl=64 time=0.054 ms

64 bytes from 172.16.2.103: icmp_seq=5 ttl=64 time=0.063 ms

64 bytes from 172.16.2.103: icmp_seq=6 ttl=64 time=0.053 ms

64 bytes from 172.16.2.103: icmp_seq=7 ttl=64 time=0.062 ms

64 bytes from 172.16.2.103: icmp_seq=8 ttl=64 time=0.071 ms

64 bytes from 172.16.2.103: icmp_seq=9 ttl=64 time=0.104 ms

64 bytes from 172.16.2.103: icmp_seq=10 ttl=64 time=0.063 ms

40G QSFP+ AOC (we tested QSFP+ DAC and AOC cables with same results)

21:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

21:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.

64 bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=1.08 ms

64 bytes from 172.16.0.6: icmp_seq=2 ttl=64 time=0.638 ms

64 bytes from 172.16.0.6: icmp_seq=3 ttl=64 time=0.628 ms

64 bytes from 172.16.0.6: icmp_seq=4 ttl=64 time=0.609 ms

64 bytes from 172.16.0.6: icmp_seq=5 ttl=64 time=1.31 ms

64 bytes from 172.16.0.6: icmp_seq=6 ttl=64 time=1.30 ms

64 bytes from 172.16.0.6: icmp_seq=7 ttl=64 time=1.31 ms

64 bytes from 172.16.0.6: icmp_seq=8 ttl=64 time=1.06 ms

64 bytes from 172.16.0.6: icmp_seq=9 ttl=64 time=1.32 ms

64 bytes from 172.16.0.6: icmp_seq=10 ttl=64 time=1.33 ms

As one can see, the latency is horrible via 40G QSFP+. Any ideas? Any help is greatly appreciated.

Thank you.

[Caguicla_Intel]

Hello cubewerk,

Thank you for the response.

We do understand where you are coming from and we sincerely apologize for the delay on this request.

Currently we are actively making a follow up to our engineers and waiting for their feedback. Unfortunately, we still do not hear anything from them reason why we can't share any progress regarding the investigation. 

We will make another follow up on this request and rest assured that we are communicating all your feedback to them. 

Thank you for your kind cooperation.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello cubewerk,

Good day!

Please be informed that we are still waiting for the feedback from our engineering team. Rest assured that we will send another update within 3 business days for the status of this request. 

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello cubewerk,

I hope you're doing good!

We'd like to inform you that we still haven't received any update from our engineers. Rest assured that we are closely monitoring this request and get back to you no later than 3 business days. 

Thank you for your kind understanding. 

Best regards,

Crisselle C.

Intel Customer Support

[cubewerk]

Crisselle,

it looks like this has no priority to your engineers. So please just be honest, if this will be solved or not, so we can move on and look for another vendor.

We need a solution!

[Caguicla_Intel]

Hello cubewerk,

We sincerely apologize on this matter as we are still waiting update from our engineering but rest assured that we are monitoring this request. We will get back to you as soon as we hear feedback from them. 

Best regards,

Crisselle C.

Intel Customer Support

[cubewerk]

Crisselle,

you dont have to aplogize over and over again if nothing changes. Just get back to us if you have a news from your engineers.

[Caguicla_Intel]

Hello cubewerk,

Appreciate your reply. 

We will take note of your request and rest assured that we are closely monitoring this request then give you an update once we heard from our engineers. 

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello cubewerk,

I hope this message finds you well!

Please be informed that engineering is still checking your request and they would like to get more information for them to further validate.

Engineering is requesting the output from the following for both of the 10G and 40G configurations:

ethtool devname

ethtool --show-fec devname

ethtool -m devname

Awaiting to your reply. 

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello cubewerk,

Good day!

This is just a follow up to check if you read our previous message. If yes, please confirm and provide the information we request. In case you need more time on this, please advise and we will adjust our next follow up.

Engineering is requesting the output from the following for both of the 10G and 40G configurations:

ethtool devname

ethtool --show-fec devname

ethtool -m devname

Hoping to hear from you soon.

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C.

Intel® Customer Support

[cubewerk · Accepted solution]

Our customer already switched to a different product and does not have the hardware anymore.