Intel XL710 40G + QSFP+ AOC/DAC-cables - very high latency

[cubewerk]

Hi,

We have 3 high-performance cluster-nodes directly connected via broadcast bond (Proxmox Linux, Kernel 5.15.35-2)

Server 1 -> Server 2

Server 2 -> Server 3

The directly connected setup works flawlessly on other clusters with 10G SFP+. The new QSFP+ 40G direct links however have a very high latency. Bond setup is simple:

iface bond0

inet static address 172.16.0.5/24

5.255.0 bond-slaves enp132s0f2 enp132s0f3

bond-mode broadcast

bond-miimon 100

Test-Results:

10G SFP+:

84:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
84:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

PING 172.16.2.103 (172.16.2.103) 56(84) bytes of data.

64 bytes from 172.16.2.103: icmp_seq=1 ttl=64 time=0.049 ms

64 bytes from 172.16.2.103: icmp_seq=2 ttl=64 time=0.056 ms

64 bytes from 172.16.2.103: icmp_seq=3 ttl=64 time=0.056 ms

64 bytes from 172.16.2.103: icmp_seq=4 ttl=64 time=0.054 ms

64 bytes from 172.16.2.103: icmp_seq=5 ttl=64 time=0.063 ms

64 bytes from 172.16.2.103: icmp_seq=6 ttl=64 time=0.053 ms

64 bytes from 172.16.2.103: icmp_seq=7 ttl=64 time=0.062 ms

64 bytes from 172.16.2.103: icmp_seq=8 ttl=64 time=0.071 ms

64 bytes from 172.16.2.103: icmp_seq=9 ttl=64 time=0.104 ms

64 bytes from 172.16.2.103: icmp_seq=10 ttl=64 time=0.063 ms

40G QSFP+ AOC (we tested QSFP+ DAC and AOC cables with same results)

21:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

21:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

PING 172.16.0.6 (172.16.0.6) 56(84) bytes of data.

64 bytes from 172.16.0.6: icmp_seq=1 ttl=64 time=1.08 ms

64 bytes from 172.16.0.6: icmp_seq=2 ttl=64 time=0.638 ms

64 bytes from 172.16.0.6: icmp_seq=3 ttl=64 time=0.628 ms

64 bytes from 172.16.0.6: icmp_seq=4 ttl=64 time=0.609 ms

64 bytes from 172.16.0.6: icmp_seq=5 ttl=64 time=1.31 ms

64 bytes from 172.16.0.6: icmp_seq=6 ttl=64 time=1.30 ms

64 bytes from 172.16.0.6: icmp_seq=7 ttl=64 time=1.31 ms

64 bytes from 172.16.0.6: icmp_seq=8 ttl=64 time=1.06 ms

64 bytes from 172.16.0.6: icmp_seq=9 ttl=64 time=1.32 ms

64 bytes from 172.16.0.6: icmp_seq=10 ttl=64 time=1.33 ms

As one can see, the latency is horrible via 40G QSFP+. Any ideas? Any help is greatly appreciated.

Thank you.

[Caguicla_Intel]

Hello cubewerk,

Thank you for posting in Intel Ethernet Communities. 

We are sorry to hear that you are having issue with your Intel XL710. For us to check the issue, please provide the following information. 

1. Driver version used on you Intel XL710

2. Will you be able to share if you see yottamark or BradyID stickers from the physical adapter? These stickers are authentication labels. This will help us identify if you are using a retail or Original Equipment Manufacturer(OEM) version of Intel Ethernet Adapter.

You may refer to link below for YottaMark* and BradyID* Stickers on Intel® Ethernet Adapters

https://www.intel.com/content/www/us/en/support/articles/000007074/network-and-i-o/ethernet-products.html

3. Can you share the brand and model of the QSFP+ AOC/DAC-cables you used?

4. Operating System used where the XL710 was installed.

Awaiting to hear from you soon. 

Should there be no reply, we will follow up after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[cubewerk]

Hi Crisselle,

driver is:

filename: /lib/modules/5.15.39-4-pve/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
author: Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion: CEFF8ACB01F180F6F7BB885

We see all 3 stickers and according to brady-website, all 3 codes show up as

We use the following 2 cables (DAC+AOC)

https://www.fs.com/de/products/36278.html

https://www.fs.com/de/products/120520.html?attribute=1691&id=196850

Operating System is Debian 11.

[Caguicla_Intel]

Hello cubewerk,

Appreciate your swift response.

Please see below information for some feedback and question that will help with the investigation. 

1. How many XL710-QDA2 adapters are affected by the issue?

2. Please be informed that Debian is not listed as support Operating System for XL710-QDA2. You may visit link below for list Supported Operating Systems for Retail Intel® Ethernet Adapters. Do you have a different system that has a supported OS listed below that you can test to further isolate the issue?

https://www.intel.sg/content/www/xa/en/support/articles/000025890/ethernet-products.html

3. Have you tried the latest driver version 2.20.12 from Intel download site below. We'd like to set your expectation that this may or may not work as the Operating System you used is not listed as validated OS for this adapter. 

Intel® Network Adapter Driver for PCIe* 40 Gigabit Ethernet Network Connections under Linux*

https://www.intel.com/content/www/us/en/download/18026/intel-network-adapter-driver-for-pcie-40-gigabit-ethernet-network-connections-under-linux.html

4. We would also like to share the list of validated Optic modules and DAC cables for Intel® Ethernet Converged Network Adapter XL710-QDA1/QDA2. Only Intel branded modules are listed from the site.

https://compatibleproducts.intel.com/ProductDetails?activeModule=Intel%C2%AE%20Ethernet&prdName=Intel%C2%AE%20Ethernet%20Converged%20Network%20Adapter%20XL710-QDA1%2FQDA2#

Awaiting to your reply.

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel® Customer Support

[cubewerk]

Hi Crisselle,

before we start switching OS,drivers, cables, can you please confirm, that in general, a ping latency lower than 100ns is possible with this card and validated intel cables?

[Caguicla_Intel]

Hello cubewerk,

Thank you for the swift response. 

Please allow us to further check your question with our engineers. We will give you an update as soon as possible but no later than 3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello cubewerk,

Good day!

Please be informed that we already escalated this request to our engineers and they are currently checking on this. Rest assured that we will give you an update as soon as we heard from them but no later than 3 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello cubewerk,

Good day and I hope this message finds you well!

We'd like to inform you that we are still actively working on this request with our higher level Engineers. We apologize for the delay and rest assured that we are doing our best to sort this out as soon as possible. We will give you another update within 3 business days. 

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[cubewerk]

We tried in the meantime the latest Intel drivers (2.20.12) with no difference.

Awaiting your response.

[Caguicla_Intel]

Hello cubewerk,

Thank you for trying out the latest driver and we are sorry to hear that it didn't help.

Our engineering team is requesting for the following information that would help with the investigation. We hope you don't mind sharing this with us for us to continue with this one. 

1. Output of ethtool -I command for both cards

2. Output of ethtool -m command for both cards with all SFP/QSFP attachments

3. Pictures of all 3 labels from both NICs. We will send an email for you to attach the photos. Please refrain on posting the photos on this thread since it contains Serial number of the card. 

Awaiting to your reply. 

Should there be no response from you, we will follow up after 3 business days. 

Best regards,

Crisselle C.

Intel® Customer Support

[cubewerk]

Pictures and ethtool-output has been provided via e-mail.

[Caguicla_Intel]

Hello cubewerk,

Thank you for providing the information requested. 

Please allow us to forward this to our engineers to continue with the investigation. Please expect another message no later than 3 business days for the status of this request. 

Thank you for your kind understanding. 

Best regards,

Crisselle C.

Intel Customer Support 

[Caguicla_Intel]

Hello cubewerk,

Good day!

We received the output from ethtool -m but it looks like you missed to provide the outpout of ethtool -I command for both cards as also requested by our engineers? We hope you don't mind sharing this with us so we can continue the investigation. 

Awaiting to your reply. 

Should there be no response from you, we will follow up after 3 business days. 

Best regards,

Crisselle C.

Intel® Customer Support

[cubewerk]

Both outputs are part of the 2 files, i have sent you via e-mail.

The files are named: n1-ethtool-output.txt and n2-ethtool-output.txt

The file starts with...

root@proxmox2-ng:~# more n2-enp33s0f0.txt
Channel parameters for enp33s0f0:
Pre-set maximums:
RX: n/a
TX: n/a
Other: 1
Combined: 64
Current hardware settings:
RX: n/a
TX: n/a
Other: 1
Combined: 64

That is the output of ethtool -l

Right after that in the same file, you find the output of ethtool -m.

Identifier : 0x0d (QSFP+)
Extended identifier : 0x00
Extended identifier description : 1.5W max. Power consumption
Extended identifier description : No CDR in TX, No CDR in RX

...

[cubewerk]

I think its a display error, that lead an "ELLL" to be displayed in this forum as an "IIIIH".

Here are the reqeusted outputs:

root@proxmox2-ng:~# ethtool -i enp33s0f0
driver: i40e
version: 5.15.39-3-pve
firmware-version: 8.70 0x8000c40f 255.65535.255
expansion-rom-version:
bus-info: 0000:21:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

root@proxmox2-ng:~# ethtool -i enp33s0f1
driver: i40e
version: 5.15.39-3-pve
firmware-version: 8.70 0x8000c40f 255.65535.255
expansion-rom-version:
bus-info: 0000:21:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[Caguicla_Intel]

Hello cubewerk,

Thank you for sharing the output of ethtool -i on this thread. 

Please allow us to re-escalate this request to our engineers to continue the investigation. We will give you an update as soon as possible but no later than 3 business days. 

Best regards,

Crisselle C.

Intel Customer Support 

[Caguicla_Intel]

Hello cubewerk,

Good day!

Please see below feedback from our engineers and feel free to let us know if you have questions or clarifications. 

1. Unfortunately, the driver version shown is not recognized. Can you confirm if this is the driver that came with the kernel?
2. Have you tried any testing with the latest official driver version 2.20.12 from https://sourceforge.net/projects/e1000/files/i40e%20stable/
3. Have you also tested the latency without the bonding?

Awaiting to your reply. 

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel Customer Support

[cubewerk]

Crisselle,

if you scroll up a bit you would see, that i already tested the latest intel kernel without any difference and informed you about the results. Please dont make it more complicated.

I also tested without a bond and a direct connection - same results.

Criselle, if intel is unable to resolve this until the end of this week, we have to return all intel equipment in this project as we can not postpone this any further.

Nearly a month has passed without your side even trying to reproduce the issue in a lab or reporting any test rsults form your site. The intel customer support was of no help so far.

[Caguicla_Intel]

Hello cubewerk,

Thank you for the swift reply.

Please accept our sincerest apologies as we didn't notice the update regarding your test on latest Intel drivers. 

Appreciate your update on testing without a bond and a direct connection. We will forward your feedback to our engineers. 

We do apologize again for the delay and inconvenience that this has caused. We understand that this request has been open for quite some time and we still haven't provided any resolution yet. Rest assured that this is being investigated by our engineering team. We will get back to you as soon as we heard from them but no later than 3 business days.

Hoping for your kind understanding.

Best regards,

Crisselle C.

Intel Customer Support

[Caguicla_Intel]

Hello cubewerk,

We sincerely apologize as we still do not have a feedback we can share from our engineering team. The issue is a complex one that we need more time to investigate. 

Rest assured that we are making a close follow up to them and we will get back to you as soon as we heard an update from them. 

Hoping for your kind understanding. 

Best regards,

Crisselle C.

Intel® Customer Support 

[cubewerk]

Crisselle,

I'm pretty certain it should be possible for an intel professionell to bump up 2 test-systems within 4 weeks and do a simple ping between the devices and provide feedback.

I'm not expecting good results/solution from your site but at least test results on your hardware to compare numbers.