Cyclone 10 LP reverting to factory image unexpectedly

[Terry6]

I have a design with both factory and application images. On power-up everything works as expected, the factory image is loaded, then the application image is loaded.

I'm using the Remote Update (RU) IP that includes the Avalon memory-mapped wrapper, so all reads/writes of the RU module are via Avalon transactions. The clock provided to the RU core is 10MHz.

While the application image is loaded I occasionally (once every 1 to 72 hours) see the FPGA re-configures itself and revert back to the factory image.

The RU module indicates that the watchdog timer expired which caused the re-config back to the factory image. The watchdog timeout is set to 1.2 seconds and I reset the watchdog timer every 100ms.

To debug this, I changed the firmware to reset the watchdog every 650ms instead of 100ms. After loading this firmware the re-config issue occurs every 10-20 seconds. This indicates to me that the watchdog resets are not occurring reliably and that some of the watchdog resets are getting missed by the RU core. The "waitrequest" output from the RU IP is never asserted, so there's no concern about the Avalon writes getting held off.

My next step was to increase the duration of the Avalon write pulse from a single 10MHz clock to 2x 10MHz clocks which seems to have completely fixed the issue.

While running with the 2-clock wide write pulse and only resetting once per 1.2 second watchdog timeout, the failure rate decreased from once per 10-20 seconds to zero in many hours of testing.

Are there timing constraints that need to be applied? Is there something in the Avalon wrapper that I can look at to determine if the Avalon write was truly accepted?

Cyclone 10 LP (10CL025)
Quartus Prime version 23.1 (SC Lite Edition)
Remote Update IP version 23.1

Thanks!
Terry

[FvM]

Hi,
can you be sure that watchdog reset is performed continuously? Using AVMM interface suggests it's performed by a soft processor. 

[Terry6]

Hi,
Yes, the AVMM interface to the RU IP is managed by a firmware-implemented state machine with its own timer that periodically writes a "1" to the RU_RESET_TIMER register.
No software or processors are involved in the RU process.

Terry

[FvM]

Hi,
I would generate an GPIO signal in parallel to the Watchdog Reset and check with oscilloscope (time width trigger) if the reset period has outliers.

[Terry6]

I added the suggested GPIO and confirmed that the firmware is resetting the RU watchdog timer at the expected rate at all times, even immediately prior to the FPGA unexpectedly re-configuring itself (as indicated by CONFIG_DONE being de-asserted).

[FvM]

Hi,
other question, did you check reconfig reason in RU status register, is it WD timeout? Reboot can be also caused by power glitch or EMI reaching NCONFIG pin.

[Terry6]

Yes, the re-config trigger conditions register always says what the User Watchdog Timer Timeout caused the re-config.

Given that I can't make this work reliably without extending the duration of the Avalon write pulse to 2 clocks and violating the requirements given in the RU IP databook figures, I decided to change the design and eliminate what I believe is a timing violation in the Intel provided Avalon wrapper around the RU IP.
I re-generated the RU IP core to eliminate the Avalon wrapper, then updated my state machine to control the reset_timer and other signals directly instead of through Avalon writes.

Testing on the new design structure is not complete, but is looking very promising.

[Terry6]

Changing my design to directly control the RU IP has fixed the issue. If Intel is interested in pursuing the potential timing issues in the AVMM wrapper I an provide more detail on the original design, otherwise, feel free to close this ticket.

Thanks!

[NurAiman_M_Intel]

Hi,

Thank you for sharing the solution with Intel community. I am glad that you can now move forward with your design.

I now transition this thread to community support. If you have a new question, Please login to ‘ https://supporttickets.intel.com’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

Regards,

Aiman