e810 cannot connect to Xilinx Versal FPGA

[esamuel]

The network adapter model on the server side is Intel E810. According to the content on page 104 of its datasheet, this network adapter can support the 100G data format sent by FPGA.

Currently, the E100GQSFP28SR4 optical module is used to interconnect the server and FPGA. The test results are as follows:
1. When the connection is normal, neither side can receive the data packets sent by the other side. The E810 side cannot detect the node, and the send and receive lights are not on.

2. The FPGA uses the loopback mode. After receiving the light and converting it into an electrical signal, the input signal of Rx is looped back internally to Tx and output to E810. At this time, the E810 side can detect the node, the receiving light flashes, and the sending light flashes when sending data packets, but it cannot receive any data packets. The server side shows that the self-negotiation failed and the network rate is not read.

3. Directly use the loopback optical module with the TxRx physically short-circuited on the hardware to plug into the server. The receiving light on the E810 server side is always on, and the sending light flashes when sending data packets. The self-negotiation is successful and the network rate can be read, but the complete data packet cannot be received. (Therefore, I suspect that the packets received by the server must be formatted based on the sent packets before they can be recognized?)

The FPGA uses CAUI-4 mode, Clause82 PCS, and can send and receive data with other FPGAs normally. The E810 server can also send and receive data with other servers normally.
But the two cannot be connected.

I would like to ask, when E810 is compatible with different 100G network modes, does the server need to make corresponding modifications in the program? For example, when connecting to CAUI-4 on the FPGA side.

[Fikri_Intel]

Hi esamuel,

Thank you for reaching Intel Community.

Based on your inquiry, CAUI is compatible with E810 on the compatibility site:

https://compatibleproducts.intel.com/ProductDetails?activeModule=Intel%C2%AE%20Ethernet&prdName=Intel%C2%AE%20Ethernet%20Network%20Adapter%20E810-CQDA1%20/%20CQDA2

Due to power requirements, the Intel® Ethernet Network Adapter E810-CQDA2 Adapter supports only 1 100Gb Long-Range LR optic module per card.

Along with that, has the FPGA side been configured for 100G too?

Regards,

Fikri

[Pintu]

Hello esamuel,

Greeting for the Day!

We are currently awaiting your response regarding the case. If you have any queries or require further assistance, please feel free to respond on the community post. We are more than happy to assist you.

Thank you for choosing Intel products and services.

Regards,

Manoranjan.

[esamuel]

Thanks for your answer Frkri and Manoranjan

We confirmed that the configuration on the FPGA side is 100G, 25.775x4, CAUI-4. The two FPGAs can send packets to each other normally. The FPGA is Xilinx Versal vm1402, The optical delivers used is Intel E100GQSFP28SR4Because E810 uses RS (528,514) mode when Clause91 is selected, and Xilinx MRMAC Ethernet IP can only use RS (544,514) when Clause91 is selected, we did not use FEC, but configured the FPGA using (MAC+PCS) mode, and turned off the E810 FEC function on the server side "advertised FEC modes: None", but it still cannot send and receive packets to each other.When only PCS is used, the FPGA uses loopback, E810 can detect the node, and the LED indicator flashes but is not normally on. When the FPGA uses FEC, Clause82 RS (528,514) will not light up E810 LED indicator even if loopback is configured.
So we guess that both ends use (MAC+PCS) is correct, but there are still other configurations that do not match successfully.

Now we are trying to use a 100G network switch as an intermediary to align the formats of both ends. It has been purchased but not yet delivered.

I would like to ask, in addition to FEC, what other possible reasons may cause the connection between the E810 and the FPGA to fail? When what conditions are met, the LED indicator of the E810 will remain on or flash?

[Pintu]

Hello esamuel,

Greetings for the day!

Based on your query, please provide the following details to proceed further.

1. Are the link configuration settings (speed, duplex mode, auto-negotiation) correctly configured to prevent connectivity issues between the Intel E810 adapter and the FPGA?

2. How do you diagnose and resolve physical layer issues (Ex. faulty cables, connectors) impacting the link between the Intel E810 and the FPGA?

3. How can compatibility issues between the ice driver, irdma firmware versions, and the FPGA's networking protocols be resolved for stable connectivity?

Thank you for choosing Intel products and services.

Best regards,

Manoranjan.

[Pintu]

Hello esamuel,

Greeting for the Day!

We are currently awaiting your response regarding the case. If you have any queries or require further assistance, please feel free to respond on the community post. We are more than happy to assist you.

Thank you for choosing Intel products and services.

Regards,

Manoranjan.

[Sachinks]

Hello esamuel,

Greetings for the day!

I hope this message finds you well.

We are still waiting for your reply.

Kindly revert back to us with the details so that we can assist you further.

Regards,

Sachin KS

[esamuel]

Hello manoranjan

Sorry to reply you so late, because there has been no progress for too long. My leader has arranged me to do other work, and some verifications have not been completed.

1. We have confirmed that the speed and duplex mode are correct. Both the FPGA and the server have been tested for data transmission and reception with similar devices. The rate is 100Gb and the transmission and reception functions are correct. However, I have not confirmed whether the auto-negotiation on the FPGA side is configured normally. The auto-negotiation on the E810 server side is turned on.

2. We tested the optical module and optical fiber, and there is no physical layer issues (Ex. faulty cables, connectors). The FPGA is placed next to the server. We use the tested physical connection in the laboratory environment to complete the connection between the FPGA and the E810.

3. The configuration information is as follows:

ice driver:

modinfo ice
filename: /lib/modules/5.15.0-107-generic/kernel/drivers/net/ethernet/intel/ice/ice.ko
firmware: intel/ice/ddp/ice.pkg
license: GPL v2
description: Intel(R) Ethernet Connection E800 Series Linux Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: CD29ED80DBDD19234D521A3
alias: pci:v00008086d0000151Dsv*sd*bc*sc*i*
alias: pci:v00008086d0000124Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000124Esv*sd*bc*sc*i*
alias: pci:v00008086d0000124Dsv*sd*bc*sc*i*
alias: pci:v00008086d0000124Csv*sd*bc*sc*i*
alias: pci:v00008086d0000189Asv*sd*bc*sc*i*
alias: pci:v00008086d00001899sv*sd*bc*sc*i*
alias: pci:v00008086d00001898sv*sd*bc*sc*i*
alias: pci:v00008086d00001897sv*sd*bc*sc*i*
alias: pci:v00008086d00001894sv*sd*bc*sc*i*
alias: pci:v00008086d00001893sv*sd*bc*sc*i*
alias: pci:v00008086d00001892sv*sd*bc*sc*i*
alias: pci:v00008086d00001891sv*sd*bc*sc*i*
alias: pci:v00008086d00001890sv*sd*bc*sc*i*
alias: pci:v00008086d0000188Esv*sd*bc*sc*i*
alias: pci:v00008086d0000188Dsv*sd*bc*sc*i*
alias: pci:v00008086d0000188Csv*sd*bc*sc*i*
alias: pci:v00008086d0000188Bsv*sd*bc*sc*i*
alias: pci:v00008086d0000188Asv*sd*bc*sc*i*
alias: pci:v00008086d0000159Bsv*sd*bc*sc*i*
alias: pci:v00008086d0000159Asv*sd*bc*sc*i*
alias: pci:v00008086d00001599sv*sd*bc*sc*i*
alias: pci:v00008086d00001593sv*sd*bc*sc*i*
alias: pci:v00008086d00001592sv*sd*bc*sc*i*
alias: pci:v00008086d00001591sv*sd*bc*sc*i*
depends:
retpoline: Y
intree: Y
name: ice
vermagic: 5.15.0-107-generic SMP mod_unload modversions
sig_id: PKCS#7
signer: Build time autogenerated kernel key
sig_key: 66:36:BA:75:E1:C2:01:89:6B:68:62:8D:3F:0A:A5:21:B5:B3:F5:C9
sig_hashalgo: sha512
signature: B8:C8:B9:C7:18:7D:B4:0E:24:59:D

rdma:

sudo rdma dev show
0: irdma0: node_type ca fw 1.54 node_guid 6efe:54ff:fe40:c608 sys_image_guid 6cfe:5440:c608:0000
1: irdma1: node_type ca fw 1.54 node_guid 6efe:54ff:fe40:c609 sys_image_guid 6cfe:5440:c609:0000
2: irdma2: node_type ca fw 1.54 node_guid 6efe:54ff:fe40:c5c0 sys_image_guid 6cfe:5440:c5c0:0000
3: irdma3: node_type ca fw 1.54 node_guid 6efe:54ff:fe40:c5c1 sys_image_guid 6cfe:5440:c5c1:0000

srcversion: CD29ED80DBDD19234D521A3

We don’t know if there is a driver compatibility problem between the FPGA and the E810. Currently, We have not found a successful connection case between the Versal MRMAC and the E810. There is only an example of successful connection between this type of FPGA and nvidia (Mellanox) MCX415A-CCAT after forwarding through a 100G switch, so I am doing other work and waiting for the switch to be compatible with different formats.

This question is just to ask what possible problems may cause the two to be unable to communicate.

thank you

[Irwan_Intel]

Hello esamuel,

Thanks for the update. Please allow some time for us to check the details and we'll get back to you once we have an update.

Regards,

Irwan_Intel