Stratix10 PCIe clock structure issue

[allen18]

Hello everyone，

   I am using quartus Pro 22.3 , device ( stratix10 1SX110HN2F43I2VG ) ,  IP core ( L-Tile and H-Tile Avalon Memory-mapped Intel FPGA IP for PCI Express 22.2.0, GEN 3X4 125Mhz.

  ug_s10_pcie_avmm-683667-666643 shows that The Intel L-/H-Tile Avalon-MM for PCI Express IP supports the Separate Reference Clock With No Spread Spectrum architecture (SRNS), but not the Separate Reference Clock With Independent Spread Spectrum architecture (SRIS).

  When I use the Common Clock Architecture (pcie reference clock come from the PC board through the Oculink interface), everything is normal and the example design can run successfully. But when I modify the pin constraints and use a 100M local clock from the FPGA board, Linux kernel will continue to print errors and the example design cannot run normally. 

  I don't think these clocks are spread spectrum clocks, what could be the possible reason for this ?

[ventt]

Hi allen18,

Thanks for reaching out.

Allow me some time to investigate your issue. I shall come back to you with the findings.

Thanks.

Best Regards,

Ven

[ventt]

Hi allen18,

Yes, Stratix 10 L-/H-Tile PCIe IP does not support SRIS, and the on-board oscillator is not used for PCIe.

Please use the SRNS or Common Reference Clock with Spread Spectrum Clocking architecture instead.

Additionally, please ensure both the RP and EP sides of the link disable the spread spectrum if using the separate clock architecture.

KDB: https://www.intel.com/content/www/us/en/support/programmable/articles/000075574.html

Thanks.

Best Regards,

Ven

[allen18]

Hi ventt,

  I think my current application scenario should be SRNS，why the on-board oscillator is not used for PCIe reference clock ?

[ventt]

Hi allen18,

Referring to the Stratix 10 Pin Connection Guidelines (pg. 9), the on-board oscillator is not used for PCIe, it is used for device configuration and transceiver calibration.

Stratix® 10 Device Family Pin Connection Guidelines: https://www.intel.com/content/www/us/en/docs/programmable/683028/current/device-family-pin-connection-guidelines.html

Thanks.

Best Regards,

Ven

[allen18]

Hi ventt,

  My reference clock comes from a crystal oscillator, which has a frequency of 100MHz , not the OSC-IN pin, OSC-IN connected to another 125MHz crystal oscillator.

  As shown in the figure, AV33,AV34,The common clock, connected to PC board via oculink. AT33,  AT34  come from a crystal oscillator on FPGA board.  AV33,AV34, This set of clocks is working normally. If I connect to AT33, AT34, Will not work properly and print errors.

  Please help investigating why the latter set of clocks is not working properly.

Thanks.

[ventt]

Hi allen18,

May I know if you are using the Altera Dev Kit? https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/stratix/10-sx.html 

Could you please share the schematic file you are referring to?

Thanks.

Best Regards,

Ven

[allen18]

Hi ventt,

I'm not using DEV kit.

I cannot provide a complete schematic file but I can confirm that both reference clocks are valid and that the PCIe pin connections are correct, because when using the first set of reference clocks, DMA data transmission and reception can be performed normally.

In UG, it can be seen that when using an independent reference clock, the slot clock configuration should be set to OFF, but I do this, the situation did not change much.

When I use the second set of parameter clocks, PCIe cannot establish a link, and ltssmstate changes between 2 and 3 several times, eventually staying at 2.

[ventt]

Hi allen18,

My apology for the delayed response.

When you used the crystal oscillator, did the PCIe link up successfully?

Yes, when using the separate clock architecture, users must set the Slot Clock Configuration to "off" in the Quartus IP Parameter. Could you also confirm that the RP side of the link has disabled the spread spectrum?

Could you please ensure that the reference clock (AT33, AT34) has 100 MHz ±300 ppm according to the PCI Express Card Electromechanical Specification Revision 2.0. The input reference clock must be stable and free-running at device power-up for proper PLL calibrations and a successful configuration.

Besides, where is the other refclk, and what are the pin assignments for refclk 0 and refclk 1 in both cases?

Thanks.

Best Regards,

Ven

[allen18]

  Hi ventt,

  I am currently unable to determine whether the issue is caused by my local environment, such as clock quality. Can you try running an Example Design in your environment ? Select PCIe avmm as the IP and set the slot clock configuration to OFF.

  Thanks.

[ventt]

Hi allen18,

Could you please share the .ip file so that I can generate the Design Example based on your settings?

Thanks.

Best Regards,

Ven

[allen18]

  Hi ventt,

  The attachment is my IP file. To ensure example design generate successfully, I don't think it's necessary to keep all settings consistent as mine. Just set the slot clock configuration to OFF and use a separate PCIe reference clock.

  Thinks.

[ventt]

Hi allen18,

Thank you for uploading the .ip file.

My apologies for the delayed response.

I have tested the Design Example generated from the .ip file you provided. I used a separate PCIe reference clock from a local source instead of the common clock method, and it successfully linked up at Gen3 x4 using Stratix 10 MX dev kit. In my environment, there is no option to enable Spread Spectrum Clocking (SSC) on the host, so I believe it is off as not supported.

Could you please share the .qsf file? Also, please ensure you follow the settings for using the separate clock architecture and meet the refclk requirements.

Thanks.

Best Regards,

Ven

[allen18]

  Hi ventt,

  Thank you for your reply.

  My qsf file is derived from the example design. Since the file was encrypted, I have copied its contents here. I believe there should not be significant differences between our QSF files.

  Our current project can proceed using a ‌common reference clock‌, so this issue does not affect the project timeline. 

  Your test results have proven that the example design can run normally on the development board, I suspect the problem might stem from either my current environment’s clock not meeting the reference clock requirements or potential PCB signal integrity issues. Could you provide the qar file for your current project? ‌ Then perhaps we can close this issue.

  Thanks.

"""

set_global_assignment -name ORIGINAL_QUARTUS_VERSION 22.3.0
set_global_assignment -name PROJECT_CREATION_TIME_DATE "11:13:13 NOVEMBER 07, 2024"
set_global_assignment -name LAST_QUARTUS_VERSION "22.3.0 Pro Edition"
set_global_assignment -name FAMILY "Stratix 10"
set_global_assignment -name DEVICE 1SX110HN2F43I2VG
set_global_assignment -name VERILOG_MACRO "ALTERA_EMIF_ENABLE_ISSP=1"
set_global_assignment -name AUTO_RESTART_CONFIGURATION OFF
set_global_assignment -name STRATIXV_CONFIGURATION_SCHEME "ACTIVE SERIAL X4"
set_global_assignment -name USE_PWRMGT_SCL SDM_IO14
set_global_assignment -name USE_PWRMGT_SDA SDM_IO11
set_global_assignment -name PWRMGT_SLAVE_DEVICE0_ADDRESS 40
set_global_assignment -name DEVICE_INITIALIZATION_CLOCK OSC_CLK_1_125MHZ
set_global_assignment -name ERROR_CHECK_FREQUENCY_DIVISOR 1

set_global_assignment -name TOP_LEVEL_ENTITY pcie_ed
set_global_assignment -name IP_FILE ip/pcie_ed/pcie_ed_DUT.ip
set_global_assignment -name IP_FILE ip/pcie_ed/pcie_ed_resetIP.ip
set_global_assignment -name IP_FILE ip/pcie_ed/pcie_ed_MEM.ip
set_global_assignment -name QSYS_FILE pcie_ed.qsys

set_location_assignment PIN_AT34 -to refclk_clk
set_location_assignment PIN_AT33 -to "refclk_clk(n)"

set_location_assignment PIN_AW32 -to xcvr_rx_in3
set_location_assignment PIN_AW31 -to "xcvr_rx_in3(n)"

set_location_assignment PIN_BB30 -to xcvr_rx_in2
set_location_assignment PIN_BB29 -to "xcvr_rx_in2(n)"

set_location_assignment PIN_AY30 -to xcvr_rx_in1
set_location_assignment PIN_AY29 -to "xcvr_rx_in1(n)"

set_location_assignment PIN_AV30 -to xcvr_rx_in0
set_location_assignment PIN_AV29 -to "xcvr_rx_in0(n)"

set_location_assignment PIN_AY38 -to xcvr_tx_out3
set_location_assignment PIN_AY37 -to "xcvr_tx_out3(n)"

set_location_assignment PIN_BB38 -to xcvr_tx_out2
set_location_assignment PIN_BB37 -to "xcvr_tx_out2(n)"

set_location_assignment PIN_BA36 -to xcvr_tx_out1
set_location_assignment PIN_BA35 -to "xcvr_tx_out1(n)"

set_location_assignment PIN_BB34 -to xcvr_tx_out0
set_location_assignment PIN_BB33 -to "xcvr_tx_out0(n)"

set_instance_assignment -name IO_STANDARD LVDS -to refclk_clk -entity pcie_ed
set_instance_assignment -name IO_STANDARD "HIGH SPEED DIFFERENTIAL I/O" -to xcvr_tx_out0 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "CURRENT MODE LOGIC (CML)" -to xcvr_rx_in0 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "HIGH SPEED DIFFERENTIAL I/O" -to xcvr_tx_out1 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "CURRENT MODE LOGIC (CML)" -to xcvr_rx_in1 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "HIGH SPEED DIFFERENTIAL I/O" -to xcvr_tx_out2 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "CURRENT MODE LOGIC (CML)" -to xcvr_rx_in2 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "HIGH SPEED DIFFERENTIAL I/O" -to xcvr_tx_out3 -entity pcie_ed
set_instance_assignment -name IO_STANDARD "CURRENT MODE LOGIC (CML)" -to xcvr_rx_in3 -entity pcie_ed
set_location_assignment PIN_AC26 -to pcie_rstn_pin_perst
set_instance_assignment -name IO_STANDARD "3.0-V LVTTL" -to pcie_rstn_pin_perst -entity pcie_ed

"""

[ventt]

Hi allen18,

Thank you for the reply.

The Design Example I generated was using a rather new version of the QPDS Pro, v24.3. I will generate the Design Example using v22.3 so that you can test it in your environment. Do you have a Stratix 10 Dev Kit to test it with? 

Please kindly wait for a while, I will attach the .qar later.

Thanks.

Best Regards,

Ven

[ventt]

Hi allen18,

My apologies for the late reply. 

I have uploaded the project .qar file. This is the Gen3 x4 design example that was generated from the .ip file you sent earlier.

Thanks.

Best Regards,

Ven

*Kindly take note that 31 Mar and 1 Apr 2025 are national public holidays, please expect a delay in response.

[ventt]

Here is the screenshot of the lspci command on the host:

 

[ventt]

Hi allen18,

May I know if you have further questions on this thread?

Thanks.

Best Regards,

Ven

[ventt]

Hi allen18,

Good day.

Please let me know if you have further inquiries in this thread.

Thanks.

Best Regards,

Ven

[ventt]

Hi allen18,

As there are no further inquiries, I will transition this thread to community support.

If you have a new question, feel free to open a new thread to get support from Altera experts. 

Thanks.

Best Regards,

Ven