Cyclone 10 LP reverting to factory image unexpectedly

[Terry6]

I have a design with both factory and application images. On power-up everything works as expected, the factory image is loaded, then the application image is loaded.

I'm using the Remote Update (RU) IP that includes the Avalon memory-mapped wrapper, so all reads/writes of the RU module are via Avalon transactions. The clock provided to the RU core is 10MHz.

While the application image is loaded I occasionally (once every 1 to 72 hours) see the FPGA re-configures itself and revert back to the factory image.

The RU module indicates that the watchdog timer expired which caused the re-config back to the factory image. The watchdog timeout is set to 1.2 seconds and I reset the watchdog timer every 100ms.

To debug this, I changed the firmware to reset the watchdog every 650ms instead of 100ms. After loading this firmware the re-config issue occurs every 10-20 seconds. This indicates to me that the watchdog resets are not occurring reliably and that some of the watchdog resets are getting missed by the RU core. The "waitrequest" output from the RU IP is never asserted, so there's no concern about the Avalon writes getting held off.

My next step was to increase the duration of the Avalon write pulse from a single 10MHz clock to 2x 10MHz clocks which seems to have completely fixed the issue.

While running with the 2-clock wide write pulse and only resetting once per 1.2 second watchdog timeout, the failure rate decreased from once per 10-20 seconds to zero in many hours of testing.

Are there timing constraints that need to be applied? Is there something in the Avalon wrapper that I can look at to determine if the Avalon write was truly accepted?

Cyclone 10 LP (10CL025)
Quartus Prime version 23.1 (SC Lite Edition)
Remote Update IP version 23.1

Thanks!
Terry

[FvM]

Hi,
can you be sure that watchdog reset is performed continuously? Using AVMM interface suggests it's performed by a soft processor. 

[Terry6]

Hi,
Yes, the AVMM interface to the RU IP is managed by a firmware-implemented state machine with its own timer that periodically writes a "1" to the RU_RESET_TIMER register.
No software or processors are involved in the RU process.

Terry

[FvM]

Hi,
I would generate an GPIO signal in parallel to the Watchdog Reset and check with oscilloscope (time width trigger) if the reset period has outliers.

[Terry6]

I added the suggested GPIO and confirmed that the firmware is resetting the RU watchdog timer at the expected rate at all times, even immediately prior to the FPGA unexpectedly re-configuring itself (as indicated by CONFIG_DONE being de-asserted).