System-Console device_load_sof SEVERE crash on Stratix 10

covfefe · ‎02-02-2023

Hello,

I am running into an issue, where trying to load an *.sof via the system-console crashes (both command line device_load_sof and GUI: Device>Program Device).

I have the same issue regardless of:

System-Console version (20.4, 21.2, 21.4, 22.3, 22.4)
hardware (tried multiple Stratix 10 GX Dev Kits)
computer (tried multiple different Windows 10 PC).
USB cables (tried multiple ones)

The board is connected directly via USB 3.0-port to a PC (no USB hub).

When the same exact *.sof is loaded via Quartus Programmer, everything works fine: there is never an error.

Here is an example of the different error messages encountered:

Error (18939): Unexpected error in JTAG server: JTAG chain broken

Error (18939): Unexpected error in JTAG server: Chain in use - check

java.lang.Exception: Unable to load the sof: C:\fpga_s10-12-1-2022b.sof
        at com.altera.systemconsole.internal.plugin.jtag.SofProgrammer.programDevice(SofProgrammer.java:106)

See also attached full logs.

RichardTanSY_Intel · ‎02-02-2023

FINE: Device has design hash 55E3CACE465FC73C6953 but design has hash 49ADE904FA4E4588D4B9, Device has different visible SLD agents

-> This sounds like the SOF on the board does not match with the SOF chosen for linking the design.

Please double check that you're using the same SOF that's programmed.

Design will be needed to debug if the issue persists. Please help to share.

Best Regards,

Richard Tan

covfefe · ‎02-03-2023

@RichardTanSY_Intel thank you for looking at this.

I can share the sof, but it is a large file, it might be better via email/conference call?

Here is the first few lines, opened as a text file:

SOF   @ Quartus Prime Version 21.2.0 Build 72 06/14/2021 SC Pro Edition   1sg280lu2f50e2vg / /  z1530a $ snap-RC4.58ND5revB0P1_DEV__17ww195a  Untitled    ÿÿÿÿ6 Ð\ ¬ sof ?Quartus Prime Version 21.2.0 Build 72 06/14/2021 SC Pro Edition"1sg280lu2f50e2vg*z1530a2#snap-RC4.58ND5revB0P1_DEV__17ww195a:Untitled@ÿÿÿÿR ²ã

You can see that the device built for is the 1SG2801U2F50E2VG, which corresponds to the Stratix 10 GX L-tile Dev Kits we have:

I have also double checked, but removing the heat sink, and looking at the part number printed on the FPGA, and see 1SG2801U2F50E2VG.

I am not sure why we get this notification, however it is not an error or warning, it is under the "fine" category.

Regardless of this message, the image *.sof still loads, about 1/4th of the time. I just have to keep trying and trying.

See log of same attempts through Quartus Programmer 22.3: same error messages on first 3 attempts, then on 4th attempts it worked. Could this be something with jtag speed? Is there a way to slow down loading the device sof on the system-console?

covfefe · ‎02-06-2023

Here are more log files of errors seen, sometime resulting in a java crash.

Feb 06, 2023 9:47:13 AM com.altera.systemconsole.internal.core.SystemConsole logInfo
INFO: Finished discovering USB connections
Info (19848): Regular SEU info => 105 sector(s), 8 thread(s), 10000000 interval time in microsecond(s)
Info (19848): IO hash is A08D0413E892EE9BEFB02EDC069509C223C74B95BC45D3232EFEBB4F109D993E
Info (19848): Keyed hash is C72FC6717073F74321069028720DCC47EE8CFD6C94E9D5F09E10E6A887598B0C
Info (19848): Design hash is D16EDFC4FC09CCE805033D7A9675C9A5B743393DEF219493D586D4388EE3DCCC
Info (19848): IO hash is A08D0413E892EE9BEFB02EDC069509C223C74B95BC45D3232EFEBB4F109D993E
Info (19848): Keyed hash is AC65047A4F4220AB72BCC70F65506F1C583D665B554F487D30475A5AA5EFFDEF
Info: *******************************************************************
Info: Running Quartus Prime Programmer
Info: Command: quartus_pgm --no_banner --mode=jtag --cable="USB-BlasterII on localhost [USB-1]" --operation="p;C:\fpga_s10-1-15-spi-lock-fix.sof@2"
Info (213045): Using programming cable "USB-BlasterII [USB-1]"
Info (213011): Using programming file C:/fpga_s10-1-15-spi-lock-fix.sof with checksum 0x47DB41BB for device 1SG280LU2F50@2
Info (209060): Started Programmer operation at Mon Feb  6 09:47:24 2023
Error (18939): Unexpected error in JTAG server: Conflict with another device - check
Error (209012): Operation failed
Info (209061): Ended Programmer operation at Mon Feb  6 09:47:24 2023
Error: Quartus Prime Programmer was unsuccessful. 2 errors, 0 warnings
    Error: Peak virtual memory: 1670 megabytes
    Error: Processing ended: Mon Feb  6 09:47:24 2023
    Error: Elapsed time: 00:00:17
    Error: System process ID: 14300
Feb 06, 2023 9:47:24 AM com.altera.systemconsole.scripting.TCLCommand execute2
SEVERE: device_download_sof: Unable to load the sof: C:\fpga_s10-1-15-spi-lock-fix.sof
Feb 06, 2023 9:47:24 AM com.altera.systemconsole.internal.core.SystemConsole logInfo
INFO: Finished discovering JTAG connections
Feb 06, 2023 9:47:24 AM com.altera.systemconsole.internal.core.SystemConsole logFine
FINE: Connection type JTAG hash NULL index null device 1
Throw DebugException: error -6
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffb24cfa840, pid=20996, tid=0x0000000000005364
#
# JRE version: Java(TM) SE Runtime Environment (8.0_212-b10) (build 1.8.0_212-b10)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.212-b10 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [DtsFrame64.dll+0xa840][thread 5364 also had an error]

#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\covfefe\AppData\Local\Temp\\hs_err_pid20996.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

covfefe · ‎02-08-2023

@RichardTanSY_Intel , would it help if I shared the *sof file? If so, please tell me how to do so, it is large (180MB).

One solution I found online was to lower the Jtag clock rate. I was able to find commands for the Quartus Programmer, but none for the system-console. How could I get the equivalent command for the system-console?

jtagconfig --setparam <cable number> JtagClock <frequency><unit prefix> 
   i.e. with only 1 single USB blaster II plugged this will be: 
jtagconfig --setparam 1 JtagClock 6M

RichardTanSY_Intel · ‎02-09-2023

I checked that there is no available Stratix 10 GX L-tile board in our farm. I don't think getting the .sof is gonna do much since .sof is incompatible with the device.

Regarding changing the clock frequency before executing commands in system console, you may checkout this wiki:

https://community.intel.com/t5/FPGA-Wiki/Executing-Quartus-command-line-tools-from-system-console/ta-p/1198127

Let me know if this works.

Best Regards,

Richard Tan

RichardTanSY_Intel · ‎02-14-2023

May I know does my latest reply helps?

Best Regards,

Richard Tan

RichardTanSY_Intel · ‎02-16-2023

As we do not receive any response from you on the previous question/reply/answer that we have provided. I now transition this thread to community support.

Best Regards,

Richard Tan

p/s: If any answer from the community or Intel Support are helpful, please feel free to give best answer or rate 4/5 survey.