What would happen if Transceiver Calibration fails?

[xytech]

Hi

We use Arial10 10AX057H3F34E2SG for both PCIe and none-PCIe XCVRs. Some questions about XCVR calibrations.

What would happen if Transceiver Calibration fails? 

As Intel UG-01143 chapter 7 (page575) writes, "Transceivers include both analog and digital blocks that require calibration to compensate for process, voltage, and temperature (PVT) variations". 

For example, Suppose XCVR reference clock is not ready when A10 is powered up. So this XCVR is not successfully Calibrated. For now POWER-ON-Calibration Failed.

Under this situation, will XCVR cannot run at all? Or it can still run with de-rated performance (worse than it should be), such as lower speed, more bit errors, etc. I guess it should be (b), but not officially confirmed.

Suppose after a while, the XCVR reference clock is available, for None-PCIe XCVRs, user indeed could start the “re-calibration” sequence, and then XCVR could normally work. Unfortunately, for PCIe XCVRs, since Altera said "PCIe link does not allow user recalibration"(UG-01143 section 7.3), is the PCIe XCVRs DEAD now? What can we do to make it work again?

To be honest, this RE-Calibration mechanism seems to be kind of Altera-Special feature. It requires stable and free-running external CLKUSR and XCVR ref clocks BEFORE FPGA powered up. No similar requirement in Xilinx competition devices such as XCKU060, which only requires calibration resistors correctly connected for proper XCVR calibration, and this means simpler hardware design.

One of the possible reason of this is said to be: Altera XCVR’s fPLL do not have enough driving strength than Xilinx’s does. We asked our supplier Cytech (one representative of intel FPGA), but received no reply.

This is why I had to post my questions and confusions here. Anyway, bleakly hope some real expert could help on this desperated topic. Thanks, and happy weekend.

[xytech]

up

[xytech]

up. hope some expert could help....

[xytech]

999

[Nathan_R_Intel]

Hie, Please check my replies to your questions below: We use Arial10 10AX057H3F34E2SG for both PCIe and none-PCIe XCVRs. Some questions about XCVR calibrations. What would happen if Transceiver Calibration fails? This could cause functional failure and performance degradation to the transceivers. Examples of functional failure is PLL or CDR cannot lock. Performance degradation increases the intrinsic noise of the transceivers impacting the jitter performance and jitter tolerance resulting Bit Errors. Also the equalization capabilities (including adaptation feature) and clock skew could be impacted if calibration fails. As Intel UG-01143 chapter 7 (page575) writes, "Transceivers include both analog and digital blocks that require calibration to compensate for process, voltage, and temperature (PVT) variations". For example, Suppose XCVR reference clock is not ready when A10 is powered up. So this XCVR is not successfully Calibrated. For now POWER-ON-Calibration Failed. Under this situation, will XCVR cannot run at all? Or it can still run with de-rated performance (worse than it should be), such as lower speed, more bit errors, etc. I guess it should be (b), but not officially confirmed. How the failure manifest could also vary between device to device. For the case of XCVR reference clock not available when A10 powered up; this could potentially cause one of the following: i. PLL cannot lock ii. PLL can lock but performance degraded. The Bitrate or speed should be the same, but intrinsic noise level could be higher. This could cause high transmitter jitter even for refclk with low phase noise. Hence, this could contribute to bit error. iii. Extremely low chances for PLL to lock with optimum performance. Suppose after a while, the XCVR reference clock is available, for None-PCIe XCVRs, user indeed could start the “re-calibration” sequence, and then XCVR could normally work. Unfortunately, for PCIe XCVRs, since Altera said "PCIe link does not allow user recalibration"(UG-01143 section 7.3), is the PCIe XCVRs DEAD now? What can we do to make it work again? For PCIe, we do not allow user re-calibration because the spec enumeration requirement. Hence for PCIE channels, the power-up calibration only starts when refclk is available. Hence, if you have a mix of PCIe and non-PCIE XCVR, the POWER-ON calibration will wait for PCIE refclk. Even other XCVR channels will not be calibrated until the PCIe refclk is available. To be honest, this RE-Calibration mechanism seems to be kind of Altera-Special feature. It requires stable and free-running external CLKUSR and XCVR ref clocks BEFORE FPGA powered up. No similar requirement in Xilinx competition devices such as XCKU060, which only requires calibration resistors correctly connected for proper XCVR calibration, and this means simpler hardware design. Your feedback to improve ease of usage of Intel devices has been taken. We will target to do better in our future devices. Regards, Nathan

[xytech]

Hi Nathan, Thanks for patient explain which cleared my confusions. That’s great help.

To be more specific on PCIe links, our application is A10 FPGA Board connected with PC Board (Win7 OS) via PCIex4 GEN3.0 links, with board-to-board connectors. PC is root complex and A10 is endpoint of PCIe. For A10 Board, configuration scheme are JTAG and ASx4 (Autonomous Mode to comply PCIe 100ms boot time requirement).

Our concern is for JTAG mode. In development process, during we downloads new configuration data via JTAG the PCIe links will surely disconnect. But after JTAG configuration is done, what can we do to re-establish and re-enumerate the PCIe connection? Within my limited experience, at this moment the FPGA’s XCVRs MIGHT be calibrated correctly because PCIe_REF_CLK from PC MIGHT still running during JTAG configuration process. However, the windows re-enumeration will not happen automatically, because PCIe enumeration process is done by BIOS when PC booting up.

Maybe we would warm-restart PC to tigger re-enumerate, but when during PC shutdown and restart interval, the PCIe_REF_CLK will surely disappear, then FPGA XCVR might stop working as a result of lost REFCLK…..it seem to be a desperate loop….or maybe my thought is wrong somewhere. How do you think? We will appreciate your insights.

It is simpler for ASx4 mode configuration. After we update the Flash on A10 Board, we just need to power down and then re-power up the whole system, which is a whole new system cold-restart process, no problem at all. PC will start and send out PCIe_REF_CLK, at the same time FPGA Periphery Image will be re-configured within 100ms and successfully calibrated, and they will connect as expected. 

Thanks very much!

[Nathan_R_Intel · Accepted solution]

Hie Yi Xiao, Please check my replies to your questions. Do let me know if you understand my explanation and need further clarification. But after JTAG configuration is done, what can we do to re-establish and re-enumerate the PCIe connection? You only require transceiver calibration to be successful once for every power cycle of the FPGA. Hence,if you perform configuration via JTAG and PCIe refclk is available, your power up calibration will start and complete successfully. After this you do not need the power up calibration for every re-enumeration. Hence, to re-establish and re-enumerate PCIe connection; you could use any other traditional method such as warm reboot (for windows) or lspci command (for linux). Maybe we would warm-restart PC to tigger re-enumerate, but when during PC shutdown and restart interval, the PCIe_REF_CLK will surely disappear, then FPGA XCVR might stop working as a result of lost REFCLK…..it seem to be a desperate loop….or maybe my thought is wrong somewhere. How do you think? We will appreciate your insights. Actually, lost of REFCLK is not going to cost the FPGA to behave incorrectly. As long as power up calibration is successful; ON and OFF of refclk (with same PPM and phase jitter - from same source) can be accepted by FPGA PLL without performance degradation. When windows PC triggers ON and OFF of refclk; it only changes the refclk phase for every cycle; having little impact on the phase jitter and no impact on the PPM or frequency. Hence, as long as FPGA is not power cycles, it can accept the ON and OFF of refclk from same source. If FPGA is power cycled, you need to run power-up calibration again. For windows restart, the FPGA is not power cycled; hence FPGA is not required to be re-configured via JTAG; also don't require power-up calibration. Windows restart will momentarily stop and restart REFCLK and re-trigger PRSTn. However, since FPGA not re-configured or power cycled, it does not need to be re-calibrated. FPGA can work as normal based on the last power up calibration. If windows restart is performed with FPGA re-configuration, then a new power up calibration is required. As for windows shutdown, if the FPGA is powered from PCIe slot; then the FPGA experiences a power cycle and required new configuration. This will require a new power up calibration. Hence, the whole process repeats. However, if FPGA powered up externally, then you don't need to reconfigure FPGA. When windows is restarted, the FPGA still can behave normally. Hence, this is the behavior of FPGA in respective to windows restart and shutdown. Regards, Nathan

[xytech]

Hi Nathan,

Thank you very much for such detailed and patient explain. I summary as bleow according to your answer and our system real conditions, and I think I unsdrtand your explanation. If there is anything wrong, pls kindly point out.

1. A10 GX PCIe only requires successful transceiver calibration ONCE for every power cycle of the​ FPGA.

2. Our FPGA board has power supply independent of PC. During JTAG update FPGA, PCIe link between PC and FPGA will disconnect. Then we will warm-restart PC （Windows7）to re-trigger PCIe enumeration. Nothing special need to be done on FPGA as long as it stays powerd-on

3. After JTAG update is done, FPGA PCIe hard IP will stay calibrated and behave normally.

Thanks again for your help!

Yi XIAO​

[Nathan_R_Intel]

Yes, your understand is correct. Hope my answers assisted you to move forward. Regards, Nathan

[xytech]

Thanks Nathan, we start layout already.