GXB stuck in calibration: some new issues

[NWein]

Last year I had an issue where my GXB block (included as part of HDMI RX IP) was stuck in calibration: https://community.intel.com/t5/FPGA-Intellectual-Property/Cyclone-10GX-transceiver-calibration-hung/m-p/1192687

I now have a new situation.  I was, after an extended battle, able to get the HDMI RX design working.  This design consisted purely of two HDMI RX channels running in a Cyclone 10GX.  The design worked on both the Cyclone 10GX dev board, as well as our own prototype hardware.

When I incorporate those same HDMI receivers into my full design, running on the same hardware, the transceivers no longer work: they appear to be stuck in calibration again.

The HDMI RX code used is identical between the two designs (it is, literally, the same code, shared between the two quartus projects.)  As far as I can tell, all the connectivity between the pins and the HDMI blocks (clocks, resets, and serial connections) are exactly the same between the two designs.

So again: two designs, both incorporating identical HDMI RX blocks connected to the same pins, running on the same hardware: one works, one doesn't.  And once again I have no idea how to determine why calibration is not finishing in the GXB block.

In the previous thread, you suggested I work with the GXB example design to see if I can get it working on my hardware.  We are past that now; I've had the HDMI working.

Given known working hardware, what should I look at?  I feel like there must be a difference between my two projects, but I've checked through them about a thousand times and haven't come up with anything.

[Deshi_Intel]

Hi,

Pls help me to understand your problem statement first.

1. Are you saying you have 2 designs (HDMI Rx example design and HDMI full design), one works while the other failed ?

• What's the failure symptom here ? rx_cal_busy signal asserted high forever ? pls elaborate
  • Does it mean on the same hardware board, you just program HDMI example design sof then it works. Then you switched to program HDMI full design sof then it failed ?
    • If this is the case then confirm is Quartus design issue
    • It's either due to design connectivity or qsf pin setting issue in your HDMI full design
    • or it also could be due to HDMI Rx design integration to full design is not done properly due to whatever reason
    • Pls double check your full design connectivity, qsf file, Quartus synthesis and fitter compilation report and compare with HDMI example design to find for any abnormality
    • Also have you tried regenerate the HDMI IP and NativePHY IP again in HDMI full design ?

1. You also mentioned two HDMI RX channels running in a Cyclone 10GX

• Does it means you have 2 HDMI Rx IP in Cyclone 10GX ?
  • Which HDMI Rx IP is facing XCVR calibration issue ? Just one or both IP ?
  • Which transceiver channel is facing calibration issue ?

Thanks.

Regards,

dlim

[NWein]

I apologize my initial question was badly lacking in details.  Here's a simplified block diagram:

There are two HDMI RX channels instantiated at the top level, and then a whole bunch of other stuff instantiated in Platform Designer.

If I comment out the Platform Designer instantiation, and its related I/O (PCIe and the DDR3 controllers) and change nothing else, then both HDMI RX channels work correctly, and can fully lock onto incoming video.

If remove the comments and include everything, the HDMI RX channels stop working, despite the fact that they shouldn't (?) be affected by the rest of the stuff.  Timing appears similar between the two builds, no obvious differences there.

Here's a Signaltap with Channel 0 connected:

And here's one with Channel 1 connected:

In both cases, all transceivers seem to be stuck in calibration.

Again, with the rest of the design removed, this all works like it's supposed to.  I'm at a loss to figure out the mechanism by which the rest of the design is impacting the transceivers.

[NWein]

And just for completeness, here's the Signaltap of channel 0 after the Platform Designer block is commented out of the top level source file (NO OTHER CHANGES).  Everything looks great.

[Deshi_Intel]

Hi,

Np and thanks for detail explanation.

Fundamentally 2 main factor that can corrupt transceiver (XCVR) power up calibration process should be either clock or reset

• For instance, XCVR channel reset control is toggling while XCVR calibration is running half way
• or the clk supply (reflkc or clkusr) is temp lost/disconnect or become noisy due to coupling noise while XCVR calibration is running half way

Question on signal_tap result :

• In ch[0] connected result, ch[0] PLL_locked asserted high is good but why HDMI 5V detect and HPD signal of ch[1] is asserted instead of ch[0] ?
• Same for ch[1] connected result, ch[1] PLL_locked asserted high is good but why HDMI 5V detect and HPD signal of ch[0] is asserted instead of ch[1] ?
• Is HDMI 5V detect and HPD signal connection so how inverted ? Any other HDMI signals that are inverted ?

Some additional debug input :

• You mentioned using 2 HDMI interface design works but after integrating into full system design (DDR, PCIe and QSYS design) failed.
• Can you gradually just add in one additional design interface at a time and rerun test to isolate which design interface added may be causing issue to HDMI Rx ?
• Once isolated which additional design interface/module is causing HDMI failure then you can further debug on the reset and clocking connection/control at your design top level file.
  • Like any clk/PLL sharing between HDMI and the additional design interface/module ? Or potential noise coupling in FPGA internal clock network ? Are they near to each another in Quartus chip planner view ?
  • Are they sharing same reset synchronizer/control design at top level design file ? Or separate reset control ? You can debug using separate reset control to see if it solved the issue ?
  • Did you accidentally reprogram some clk generator chip on your board ?
  • Can you add further delay in your higher application software design in your overall board reset control mechanism ?

Thanks.

Regards,

dlim

[NWein]

I've since tried a great many different things, all to no effect.  Here are some details:

1. CLKUSR and the 100 MHz clock inputs should all be stable.  Nothing much happening there.
2. Yes, I had a problem with my HPDN and IN_5V_POWER signals.  I've corrected that now, makes no difference.
3. I've updated my reset logic so that a controlled 50 ms startup reset pulse is fed to the HDMI blocks, and everything else gets a 200 ms startup reset.  That should give the transceivers plenty of time to calibrate before all the rest of the logic starts up.  How long does calibration normally take?
4. The clock generators on the board are all correct.  I can freely switch between the two FPGA builds; on
5. The 100 MHz free-running clock input is fed to (a) a PLL to generate my various system clocks, and (b) the fr_clk input of the HDMI block, that's it.  As far as checking things in Chip Planner... I have a terrible time with that tool, if you can provide me more detailed instructions what to do I'd be happy to check all that.
6. I can try adding blocks one by one in Platform Designer, although that is quite tedious and will take me a while.

[Deshi_Intel]

Hi,

Below debug plan is tedious but it's also the most effective one to help you isolate for potential issue

• I can try adding blocks one by one in Platform Designer, although that is quite tedious and will take me a while.

Do you mind to share your full design with me in forum private message then I can help to take a look as well ?

Thanks.

Regards,

dlim

[NWein]

>Do you mind to share your full design with me in forum private message then I can help to take a look as well ?

I would be happy to. 

[Deshi_Intel]

alright, I will wait for you to share design with me. Thanks.

[NWein]

PMs sent.

[Deshi_Intel]

Got it

[Deshi_Intel · Accepted solution]

HI,

Your Quartus design is complicated and I may need more time to slowly digest it.

Anyway, I have below 2 extra finding to share with you

1. Your Quartus design "device and pin option" setting now set config clock source to internal oscillator instead of CLKUSR pin. This is not recommended as per below KDB guideline

• https://www.intel.com/content/altera-www/global/en_us/index/support/support-resources/knowledge-base/hsio/2019/can-i-use-the-internal-oscillator-setting-of-the-configuration-c0.html

1. This 2nd finding here could explained why HDMI transceiver channel calibration stuck after you added in "ehive_dual_pd" platform designer that contains PCIe hard IP

• https://www.intel.com/content/altera-www/global/en_us/index/support/support-resources/knowledge-base/ip/2016/why-doesn-t-the-gen3-stratix-v-pcie-hip-start-flow-control-initi.html
  • You can try remove PCIe IP design module and rerun testing to check if PCIe IP is the culprit ?

Thanks.

Regards,

dlim

[NWein]

Thanks, I will look into both of these issues when I’m back at work on Monday.

[Deshi_Intel]

HI,

Just want to follow up is there any latest finding that you want to share with me ?

Thanks.

Regards,

dlim

[Deshi_Intel]

HI,

Just wonder are you back from your holiday as I still haven't hear back from you after few weeks ?

Thanks.

Regards,

dlim

[Deshi_Intel]

HI,

I have not hear back from you for close to 1 month.

Therefore, I am setting this case to closure first.

Feel free to post new forum thread if you would like to resume the debug discussion in future.

Thanks.

Regards,

dlim

[NWein]

Hi,

Sorry I lost track of this for a while.

Getting the PCIe clocks going cleared my calibration problem, and the accumulation of other suggestions has me running successfully at 4K60. 

I am seeing some other weird issues but if I can't figure them out I'll post separately.

Thanks a million.