Agilex 7 E-Tile 100G Ethernet Bringup Issue

[vika]

Hi,

I'm working with an Agilex 7 AGFB014R24A2E2V sitting on a development board with 2 x QSFP28 ports.

Now I'm trying to bringup a 100G Ethernet link between the FPGA and our IXIA test instrument. I'm using the E-Tile Hard IP for Ethernet and E-Tile CPRI PHY Intel® FPGA IPs. I have configured it for Single 100G without FEC and instatiated it in Platform Designer so that I have 2 instatiations, one for each 100G port and JTAG Avalon Master to get access to the Ethernet and Transceiver PHY toolkits and register readouts. I have a refclk of 156.25MHz and the TX PLL clock o_clk_pll_div64 is driving both i_clk_tx and i_clk_rx. So far there is no custom logic connecting to the Avalon streaming interfaces to transmit/receive any data on the MAC side.

So problem is a can't get the link up between FPGA and IXIA. I'm using QSFP28 100GBASE-SR4 modules to connect the ports with MMF-fiber.

FPGA => IXIA direction:

Ethernet Toolkit reports TX PCS Ready: up/green light and I can see (via SignalTap) that tx_lanes_stable is asserted. However, IXIA reports link down (red light).

IXIA => FPGA direction:

Ethernet Toolkit reports RX PCS aligned: down/red light and I can see (via SignalTap) that rx_am_lock, rx_block_lock, rx_pcs_ready are all deasserted.

I tried to run the PMA external adaption in Ethernet toolkit to optimize the PMA settings to see if that would get the RX PCS to get aligned.  I also tried launching the Eye diagram viewer (in the Transceiver PHY Toolkit) but then I get the following error: eye viewer failed to show eye data: index 0 out of bounds for length 0.

Another thing I tried was to connect the 2 x FPGA ports together (using same QSFP28 100GBASE-SR4 modules). In this case the RX PCS aligns on both ports and rx_am_lock, rx_block_lock, rx_pcs_ready are all asserted.

So it seems connecting the 2 x FPGA ports works fine, but when moving one of them to our IXIA port link does not come up.

Do you suspect that my PMA settings need to be optimized to get the link up? PMA external adaption does not seem to do the job. Do you recommend using Eye diagram to do this? Do you know how to get my Eye diagram viewer to work?

One interesting observation I did while troubleshooting was when reading register 0x40080 in PMA Control and Status Register Map (found in E-Tile Transceiver PHY User Guide). For all 4 x transceivers I read 0x40080 = 0x 00030001 =>

tx_transfer_ready asserted

rx_transfer_ready asserted

tx_ready deasserted

rx_ready deasserted

rx_is_lockedtodata asserted

So this suggests that tx_ready/rx_ready from reset controller are not asserted. I reviewed my reset sequence that I perform for E-Tile Hard IP for Ethernet and E-Tile CPRI PHY Intel® FPGA IP, by asserting i_csr_rst_n, i_reconfig_reset. And after that also asserting i_rx_rst_n and i_tx_rst_n. I don't see any issue with reset sequence.

In table 57 in E-Tile Transceiver PHY User Guide I saw a comment that tx_ready/rx_ready might not get asserted if reset sequence not followed. But for me I'm doing the reset through the E-Tile Hard IP for Ethernet and E-Tile CPRI PHY Intel® FPGA IP so it should be covered by that reset?

Would you suspect that tx_ready/rx_ready issue might be related to my initial issue with link not coming up when connecting to IXIA?

Thanks in advance for you support!

[ZiYing_Intel]

Hi,

We sincerely apologize for the inconvenience caused by the delay in addressing your Forum queries. Due to an unexpected back-end issue in our system, your Forum case, did not reach us as intended. As a result, we have a backlog of cases that we are currently working through.

Please be assured that we are doing everything we can to resolve this as quickly as possible. This will take some time, and we appreciate your patience and understanding during this period of time. Your case will be attended by AE soonest possible.

Thank you again for your patience and understanding, and we are committed to provide you with the best possible support.

Best regards,

zying

[ZiYing_Intel]

Hi,

"So it seems connecting the 2 x FPGA ports works fine, but when moving one of them to our IXIA port link does not come up."

For this point, do you mean that when just connect to FPGA, it was link up and there is no issue on it? The link up issue was happen after the IXIA connection?

Best regards,

zying

[ZiYing_Intel]

Hi,

Is there any update from your side?

Best regards,

zying

[vika]

Hi Zying,

Thanks for your reply!

So it turned out that the IXIA instrument we were using had FEC enabled by default. This was not easy to see or understand since the license for IXIA that we have does not include any Layer0/Layer1 functionality, but only Layer2 configuration. So I just made a guess that maybe the FEC is enabled by default, reconfigured my E-Tile Hard IP for Ethernet and E-Tile CPRI PHY Intel® FPGA IP to enable FEC and tried and the link came up. This also explains why my FPGA-to-FPGA connection came up fine, but not the FPGA-to-IXIA connection.

Maybe it's industry standard to always have FEC enabled on your 100G Ethernet link, but at least I was not aware of this.

Thanks for your support, and I'll be back with new posts if questions arise!

Best Regards,
Viktor

[ZiYing_Intel]

Hi,

Thanks for using community support and I am now close the case since the root cause has been found. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread.

Best regards,

zying