Arria 10 GX - Problem with PCIe and 2 transceivers (1250Mbps SFP)

[AHX]

Hello community,

I'm facing a strange problem with two different Arria10 cards.

Both devices are PCIe cards with 2 SFP (small form-factor pluggable) interfaces. Both SFPs are working in a time sensitive network (TSN), so I can't change the transceiver configuration.

When I synthesize the project with only one SFP (one transceiver pair), PCIe and the SFP work fine. But when I build the project with 2 SFPs (two transceiver pairs), PCIe forces each computer to reboot and freeze. Good news: At least both SFPs work in this configuration.

There are no warnings from Quartus and I can't extract any problems when I "signaltap" the device.

I've tried synthesizing with Quartus Prime Standard 18.1.0 and 23.1.1.

I have no idea what's causing this.

Any suggestions on how to debug this?

Devices:

- 10AX048E3F29E2SG – 12 HSSI – 1 PCIe

- 10AX066H2F34E2SG – 24 HSSI – 2 PCIe

PCIe configuration

- Gen1:x4, Interface 64 bit, 125 MHz

- Native endpoint

SFP/transceiver configuration

- ATX PLL

-- GX clock output buffer

-- PLL output frequency 625 MHz

-- PLL reference clock frequency 125

-- Include master clock generation block

-- Enable bonding clock output ports

-- Enable feedback compensation bonding

-- PMA interface width 10

Transceiver Native Phy

- 1250 Mbps TX/RX duplex

- PMA and PCS bonding

- Synchronous state machine

[AHX]

Just giving this a gentle bump. Any assistance or input would be wonderful.

[AHX]

Just giving this a gentle bump again.

[RongYuan]

Hi,

We sincerely apologize for the inconvenience caused by the delay in addressing your Forum queries. Due to an unexpected back-end issue in our system, your Forum case, did not reach us as intended. As a result, we have a backlog of cases that we are currently working through.

PCIe forces each computer to reboot and freeze

This happens when you program the sof to the FPGA?

The freeze means your host cannot boot successfully? Linux or Windows?

What's the max power consumption of your A10 board? Your board requires external power supply? 

Regards,

Rong

[AHX]

Hi,

>> The freeze means your host cannot boot successfully? Linux or Windows?

General: Linux, ComExpress-PC*, power consumption (idle) without a PCIe card: 12V @ 0.80A = 9.6W
FPGA: 10AX066H2F34E2SG

Setup #1: FPGA PCIe card inserted
-- Large FPGA image (a lot of active components)
-- Gen1:x4, Interface 64 bit, 125 MHz, native endpoint
-- 2 SFPs (2 transceiver pairs)
-- Total consumption: 12V @ 2.15A = 25.92W
-- FPGA PCIe card consumption: 12V @ 1.35A = 16.2W
-- Result:
-- PC tries to boot second time then goes into some weird "sleep" mode, network traffic stopped, blank screen
-- FPGA PCIe card: Both SFPs still work (transceivers work, my network switch receives packages)

Setup #2: FPGA PCIe card inserted
-- Small FPGA image (removed a lot of components)
-- Gen1:x4, Interface 64 bit, 125 MHz, native endpoint
-- 1 SFP (1 transceiver pairs) !!!
-- Total consumption: 12V @ 1.98A = 23.76W
-- FPGA PCIe card consumption: 12V @ 1,18A = 14.16W
-- Result:
-- PC boots, SFP works, PCIe works, ...

Setup #3: FPGA PCIe card inserted
-- Small FPGA image (removed a lot of components)
-- Gen1:x4, Interface 64 bit, 125 MHz, native endpoint
-- 2 SFPs (2 transceiver pairs)
-- Total consumption: 12V @ 1.99A = 23.88W
-- FPGA PCIe card consumption: 12V @ 1.19A = 14.28W
-- Result:
-- PC boots, SFP works, PCIe does NOT work (not even listed by lspci), ...

>> This happens when you program the sof to the FPGA?

What exactly should I test here? Programming during the boot phase or when the OS is already running?

I don't know if this is important, but we are using the MT25QU512 flash chip (ASx4 mode).

* Also tested this with a normal Dell consumer PC, same behavior.

Edit: I have attached a picture of the device. 

[RongYuan]

Thanks for the info. 

In "Setup #3: FPGA PCIe card inserted", please check if you can see PCIe after a host warm reboot. 

Please provide the qsf file. Thanks.

Regards,

Rong

[AHX]

Hi,

>> "... please check if you can see PCIe after a host warm reboot. "

The card is still not visible.

>> QSF

See attached file, I needed to change the file name from QSF to RTF because of this error:

The file type (.qsf) is not supported. Valid file types are: 7z, ai, bmp, bz2, c, cdr, cpp, css, csv, cxx, dib, dmp, doc, docx, eml, f, f90, for, gif, gz, gzip, h, i, ico, icproj, img, jfif, jpe, jpeg, jpg, log, mdb, mdl, mov, mp4, odp, ods, odt, pdf, pjpeg, png, pps, ppsx, ppt, pptx, psd, qar, rar, rtf, rtx, sel, sig, sln, tar, tbz2, tgz, tif, tiff, txt, txz, v, vcproj, vcxproj, vfproj, vhd, vtt, wks, xcf, xls, xlsx, xml, xz, zip, 3g2, dmg, mp3, msi, mts, m2ts, ogg, srt, tar.gz, tar.gz.sig, tar.xz, p4, p4pp.

[RongYuan]

Thanks for the files.

Please refer to XCVR guide 1. Arria® 10 Transceiver PHY Overview

Your PCIe rx/tx pins are using bank 1C and 1D therefore you should use either PIN_AB28 or PIN_AD28 as PCIe refclk. Current PIN_Y28 can work as PCIe refclk only when PIN_AB28 and PIN_AD28 are not used.

Regards,

Rong

[Mare]

Hi. I have another question regarding the connection of reference clocks for the transceiver banks. I checked Arria® 10 Transceiver PHY Overview and Arria® 10 and Cyclone® 10 GX Avalon® Memory-Mapped (AvalonMM) Interface for PCI Express* User Guide and I can't find information on how to connect PCIe reference clock. Can you point me to the manual chapter or page number where can I find this info?

From your reply, I can make the following conclusion: Use REFCLK....CHB if first GXBL channels are used (as we have PCIe lanes 2 and 3 on bank 1D channels 0 and 1) or use REFCLK...CHT where the last two channels are used (as we have PCIe lanes 0 and 1 on bank 1C channels 4 and 5).

Kind regards, Marko

[AHX]

Hi,

First of all: Thanks for your reply. I've changed our design and was able to synthesize it. Now we need to do some heavy hardware modification on our card to test this.

Follow up question: As already written we're facing this problem on a different FPGA (different bank layout).

Could you please tell me how to connect the clocks? I'm attaching our QSF file (renamed to rtf).

[RongYuan]

Similarly, your pcie_refclk for this should be either PIN_R24 or PIN_U24. Current location PIN_N24 is not recommended.

Regards,

Rong

[Mare]

Hi, Are there any restrictions when mixing two different interfaces on the same Transceiver bank? For instance PCIe and two SFPs. Suppose we take  Bank 1C and 1D and put on it PCIe x4 which is split between both. Where should we put an additional two SFP interfaces (each SFP uses one RX and one Tx pair plus one dedicated clock input).