PCIe hard IP Bringup on Cyclone 10 GX custom board

[jpang10]

Hello,

We are having trouble bringing up the PCIe Hard IP on our custom board. The hard IP is set up for gen 1 x1. It appears that coreclkout is stuck low and not toggling.

Current setup:

We are using a FPGA project based on 
https://fpgacloud.intel.com/devstore/platform/18.0.0/Pro/cyclone-10-gx-pcie-gen1-x1-avl-st/

The only changes we made are pin assignments for the general IO to match our custom board, and using Quartus Pro 20.4.0.

The custom board uses the Cyclone 10 GX and the hardware is set up for up to 2 lanes. The TX and RX lanes location have been verified to be correct using pin planner. They are located at bank 1C ch4 (lane0) and 5 (lane1). We also verified that the refclk is 100MHz coming from another custom board generated from a CDCM9102RHBT clock generator. 

nPERST is controlled by the root complex and located at the dedicated pin (NPERSTL0) with VCCIO at 1.8, and we can see the signal goes high using signaltap

The pcie RX lane and refclk are set as CML I/O standard, and the TX lane is set at "high speed differential IO"

From our root complex, we can see that the lttsmstates changes from 0x00 > 0x01 > 0x06 > 0x01 > 0x02 > 0x03 and it stays there.

From the Cyclone 10 GX endpoint side using signaltap, the coreclockout/pld_clk, lttsmstates, and currentspeed are all stuck low.

Any suggestions on how we can go about debugging this would be greatly appreciated, or possible reasons why coreclkout is stuck low.

Thank you

[SengKok_L_Intel]

The coreclkout is derived from the PCIe Refclk, below are the posibilities of this problem:

1. How do you connect the npor signal? Please ensure this signal is asserted or you may just tie it as "1" in the design.
2. You may try to use the PCIe refclk as the sampling clock in signal tap and check if the serdes_pll_locked signal can be asserted? If not, it could be something wrong with the refclk.

Regards -SK

[jpang10]

Thank you for your reply, SK!

1. npor is (user_input_nrst & perstn), from signal tap it shows that it goes high. I also tried tying it straight to 1'b1, but the lttsmstates shown on the root complex are still the same

2. I used the PCIe refclk as the sampling clock, and do see serdes_pll_locked goes high. I also added the coreclkout to the signaltap, but doesn't see it switching.

Do you have any suggestions on what else we can look at?

Thanks!

[SengKok_L_Intel]

You may not see the "coreclkout_hip" switching as the sampling clock 100Mhz may be running slower. You can create a free-running counter in your design and clock by "coreclkout_hip" if the counter keeps increasing, which means the coreclkout_hip is alive. When the serdes_pll_locked is asserted, you should see the "coreclkout_hip" is running at the expected frequency.

[jpang10]

Hi SK,

We added a counter for coreclkout_hip, but the counter stayed at 0.

We signaltap these signals using pcie_refclk as the sampling clock:

pld_clk_inuse = 0

serdes_pll_locked = 1

pld_core_ready = 1

reset_status = 1

perstn = 1

npor = 1 (hard tied to 1)

ltssmstate = 0

We also tried setting a signaltap with sampling clock coreclkout_hip, but the signaltap status is "waiting for clock".

When we were watching the pcie trainings on youtube, the powerpoint said to set refclk input to CML or HCSL, but on the device datasheet, it doesn't provide the I/O standards specifications like it does for the other standards (LVDS, RSDS, LVPECL, etc). Would you know the IO standards (like Vcm, VID, etc) for CML and HCSL for Cyclone 10GX?

Since we use the CDCM9102 clock generator to create the pcie refclk, we followed figure 9 (interfacing between LVPECL and HCSL) of the datasheet for the input of the refclk. Do you see any issues with setting the pcie_refclk inputs as HCSL and following that figure to shift LVPECL to HCSL?

Do you have other suggestions that we can look into?

Thank you so much for your help

[SengKok_L_Intel]

Hi,

Most of the use cases of PCIe will share the reference clock with the host via the PCIe edge finger. Do you mean your design requires using a separate clock source between the Host and Endpoint? Besides, as per the C10 pin connection guidance, the DC coupling is allowed for PCIe reference clock if the IO standard is selected is HCSL. Please ensure you are using DC coupling as per requirement.

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/dp/cyclone-10/pcg-01022.pdf

Does the pld_clk is connected with coreclkout_hip in your design? Next, you probably can create a simple design, and input the 100Mhz to an fPLL, and see if you can get the 62.5Mhz/125Mhz output.

[jpang10]

Thanks SK!

Sorry for the confusion. The refclk does come from the golden fingers for the Cyclone 10GX, but is generated with CDCM9102 on the host side. This clock generator output is set to LVPECL, so we were following the datasheet to convert it to HCSL. We could also set the output to LVDS.

The pld_clk is connected to coreclkout_hip in the design.

For the simple project suggested:

I used the "fPLL Intel Arria 10/Cyclone 10 FPGA IP" selecting "CORE" for FPLL mode, and the pll isn't locking or outputting. The pll uses pcie_refclk (REFCLK_GXBL1C_CHBN) as input, and it's set to output 125MHz.

Could this explain why my coreclkout_hip was low and ltssm states aren't changing? 

Just for some clarification, should pll_powerdown be 1 or 0 (I tried both and it still didn't work)? Does 1 mean the fpll is powered down and output = 0?

The signal tap uses the pcie_refclk as sampling clock and it appears to work since I can monitor other signals.

I plan to look into my input logic levels for the refclk, but if there are other suggestions of what to look into, please do let me know

[SengKok_L_Intel]

The pll_powerdown signal is high when you wanted to reset the PLL. You may do a reset after power on, for example, hold the fPLL at reset for some time, and then release the reset. Below is the example code you can use, where the clk_lock signal can connect to pll_powerdown.

 reg [9:0] count_rst =0; 

 assign clk_lock = count_rst[9];

 always @(posedge clk_100MHz) begin 

if (count_rst[9]!= 1)

count_rst = count_rst +1;

else 

count_rst = count_rst;

end

Besides, please try to use the IOPLL Intel FPGA IP and see if you also encounter the same problem.

Here is the Cyclone 10 GX Pin connection guideline, please review your custom board to ensure it meets the requirement:

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/dp/cyclone-10/pcg-01022.pdf

[jpang10]

Hi SK, 

Here's the results from some of the recommendations to try:

I've tried holding the PLL pins in reset for longer period of time after power up, but that didn't make any difference to the results.

Using REFCLK_GXB (100MHz) as source:

IOPLL works: PLL locks, counter runs, and can be measured with oscope

fPLL doesn't lock, counter stays at 0

Using regular clk pin (50MHz) as source:

IOPLL works: PLL locks, counter runs, and can be measured with oscope

fPLL: not allowed

Using the output of IOPLL (100MHz) as source: 

fPLL doesn't lock, counter stays at 0

I verified with the pin connection guide:

VCCR_GXB = VCCT_GXB = 0.95V

REFCLK_GXB : currently set as CML input, AC coupled with 0.1uF and 100 ohm termination

unused REFCLK_GXB : are tied to ground individually

CLKUSR : we have a free running 100MHz oscillator supply to that pin (in fitter report, Automatically reserve CLKUSR pin for calibration purposes is ON)

RREF_[T,B] : are individually tied to a 2kΩ resistor to gnd

Thank you so much for your help so far! Do you have other suggestions of what to look into or possible reasons why FPLL isn't functioning?

[SengKok_L_Intel]

In order to ensure we are aligned, I created a simple fPLL design, please find it in the attachment.

I verified this design in A10 GX setup, what you need to do is to change the device to C10 GX, pcie refclock pin, and maybe upgrade the fpll. This design is working fine on my side by checking from the signal tap.

[jpang10]

I tried using your project as is, using Quartus Pro 20.1 and 20.4, with just upgrading the IP, and assigning the input clock pin assignment.

pll_cal_busy = 1

pll_locked = 0

counter = 0183h (not changing)

In addition to trying your project, I added a iopll and a 2nd counter for the iopll output. The iopll is working, with iopll_locked = 1 and counter2 incrementing. 

I also uploaded 2 projects. One using the project you uploaded in the previous message with just an upgraded fpll, and pin assignment. The 2nd one is with an additional iopll in the project, which is used to generate the pictures above.

Thank you! 

[SengKok_L_Intel]

To identify if there is FPLL dependency, could you please change the Fpll location by using the following setting in the QSF file?

set_location_assignment FPLL_1CT -to inst|xcvr_fpll_a10_0|fpll_inst

or

set_location_assignment FPLL_1DB -to inst|xcvr_fpll_a10_0|fpll_inst

Besides, how many boards that you encounter this problem?

Regards -SK 

[jpang10]

Hi SK,

I've tried all 4 locations (1CT, 1CB, 1DT, and 1DB) of fpll, and verified that they were placed in the specified location using the chip planner after compiling. FPLL still doesn't output in any of the location, while IOPLL does.

I have a total of 3 boards, and have tried on 2 of them, and they behave the same way. IOPLL works, and FPLL doesn't. 

I really appreciate all the help so far. Do you have other suggestions of what I can look into?

[jpang10]

Hi SK,

My FPLL works after I disabled calibration using 
set_global_assignment -name VERILOG_MACRO "ALTERA_XCVR_A10_DISABLE_RESET_CONNECTED_TO_CAL_BUSY=1" 

Would there be an issue if calibration is disabled? It appears that calibration was busy, holding my fpll in reset.

What could be some possible causes that calibration wasn't able to complete?

[SengKok_L_Intel]

Do you see the coreclkout is still stuck low if using this workaround?

Regards -SK

[jpang10]

Yes, I still see coreclkout_hip stuck low after trying this workaround

serdes_pll_locked = 1

coreclkout_hip = 0

[SengKok_L_Intel]

Can you attach a simplified PCIe design that can replicate the problem here? I would like to do a sanity check on your design. Thanks.

[jpang10]

This project is based on https://fpgacloud.intel.com/devstore/platform/18.0.0/Pro/cyclone-10-gx-pcie-gen1-x1-avl-st/

I added a separate fpll to the project and that is working (using a counter in signaltap). However, the coreclkout_hip is still stuck low (also have a counter).

Debug1-4 are just LEDs on the board.

Please let me know if you see any problems. Thank you!

[jpang10]

We have a separate question, is there a way to set up the hard IP to do a simple loopback? Can we simply connect the tx_st and rx_st together?

I see a lot of examples with the Example Application included in the platform designer. 

We currently want to test if we can complete lane training with our root complex, but the coreclkout_hip staying low could be one of our problems.

[SengKok_L_Intel]

Hi, 

No, PCIe IP can't do a loopback test. What you can do is to using the Native PHY IP to achieve that. 

I attached two SOF files, could you please capture the signal tap for each SOF, and send it back to me? I would like to see if there is any abnormality. 

[jpang10]

Thanks SK!

I figured out the coreclkout issue. It was my oscillator to the clkusr pin not outputting the 100MHz properly, so the transceiver never comes out of "busy". After making a small modification on my board to fix the oscillator, the coreclkout_hip outputs fine now and we can see signals toggling. I attached screenshots of the signaltap attached in your previous response.

However, my ltssmstate is toggling between 0x00 (Detect.Quiet), 0x01(Detect.Active), 0x02(Polling.Active). Do you have some suggestions on what could be causing this?