RiscFree debugging fails after first debug

[BrendenB]

I have a simple Nios V system that prints "Hello World" on a DE-10. The design is generated using Quartus Std Lite 23.1.1. The first time I try to debug, everything is fine. The second time I try to debug, Ashling returns and error when a try and add a break point:

"Operation Failed: Command aborted"

Looking at the GDB output I get:

Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0x6a8

The only way I can recover is to program the .sof again, but it only works for one debug.

Is there anything that I can do because its annoying to have to program the SOF every time I want to debug?

Notes:

There is something strange in the at the ash-riscv-gdb-server output. On the first (successful) run it looks like:

Ashling GDB Server for RISC-V (ash-riscv-gdb-server).
v23.4.1, 03-Nov-2023, (c)Ashling Microsystems Ltd 2023.

Initializing connection ...
Checking for an active debug connection using the selected debug probe (SN: 1):
Connected to target device with IDCODE 0x2d020dd using USB-Blaster-2 via JTAG at 16.00MHz.
Info : Active Harts Detected : 1
Info : [0] System architecture : RV32
Info : [0] Number of hardware breakpoints available : 1 
Info : [0] Number of program buffers: 8
Info : [0] Number of data registers: 2
Info : [0] Memory access -> Program buffer
Info : [0] Memory access -> Abstract access memory
Info : [0] CSR & FP Register access -> Abstract commands

Waiting for debugger connection on port 3333 for core 0.
Press 'Q' to Quit.

On the second (failed) an subsequent runs it looks like:

Ashling GDB Server for RISC-V (ash-riscv-gdb-server).
v23.4.1, 03-Nov-2023, (c)Ashling Microsystems Ltd 2023.

Initializing connection ...
Checking for an active debug connection using the selected debug probe (SN: 1):
Connected to target device with IDCODE 0x2d020dd using USB-Blaster-2 via JTAG at 16.00MHz.
Info : Active Harts Detected : 1
Info : [0] System architecture : RV32
Info : [0] Number of hardware breakpoints available : 1 
Info : [0] Number of program buffers: 8
Info : [0] Number of data registers: 2
Info : [0] Memory access -> System bus (Data width : 128 Address size : 57)
Info : [0] Memory access -> Program buffer
Info : [0] Memory access -> Abstract access memory
Info : [0] CSR & FP Register access -> Abstract commands

Waiting for debugger connection on port 3333 for core 0.
Press 'Q' to Quit.

It looks like it has detected Memory access -> System bus (Data width : 128 Address size : 57) as a way to access memory. This seems strange because it didn't exist on the first run. Also the values for the width + address size are just wrong (and can change between runs).

This also happens if I use ash-riscv-gdb-server from the command line outside the RiscFree IDE. I have also tried using OpenOCD. It never fails. However, it causes ash-riscv-gdb-server to fail in the same way when its executed after OpenOCD.

Externally I was also able to debug multiple times provided I don't shutdown the ash-riscv-gdb-server. Everything starts failing when ash-riscv-gdb-server is re-run.

[JingyangTeh_Altera]

Hi

After each debug did you close and remove the previous debug connection in the IDE ?

Regards

Jingyang, Teh

[BrendenB]

I hit the "stop" button which exits the debugger  and ash-riscv-gdb-server.

I'm not 100% sure what you mean by removed, but I never deleted the run configuration. I just hit "stop", waited 15 seconds, then hit "debug" again.

I was also able to reproduce the error using the cli directly. The steps:

1. Program the .sof to start in clean state using: quartus_pgm --no_banner --mode=jtag -o "p;$PWD/../output_files/top.sof@2".
2. Run the ash-riscv-gdb-server: ~/intelFPGA/23.1.1/riscfree/debugger/gdbserver-riscv/ash-riscv-gdb-server --probe-type usb-blaster-2 --transport-type jtag --device 0x02D020DD --auto-detect true --jtag-frequency 24 --gdb-port 3333. This server starts without Memory access -> System bus.
3. Connect to the server using the debugger: ~/intelFPGA/23.1.1/riscfree/toolchain/riscv32-unknown-elf/bin/riscv32-unknown-elf-gdb -ex "target remote :3333" -ex "load" -ex "break main" -ex "c" app/build/Debug/hello.elf
4. You can connect to the server as many times as you want and it will work provided you don't shut it down.
5. Killed the server with 'Q'.
6. Run the ash-riscv-gdb-server with same command line above. The server starts with the Memory access -> System bus.
7. Connect to the server using the debugger with the command above. End up getting the message "Cannot insert breakpoint 1....Cannot access memory at address 0x6b0".

I added a bad_niosv.txt of all the output above.

I also did some digging. The debugging spec seems to give three different ways of letting the debugger access memory.

1. System bus
2. Program buffer
3. Abstract access memory

On the first run only Program buffer and Abstract access memory are available. The System bus method does not appear to be implemented by the Nios V (just guessing here). The System Bus Access Control and Status (sbcs, at 0x38) from the debugging spec reports if the System Bus method is supported. It would seem that this register is changed between the two invocations of the ash-riscv-gdb-server. I'm assuming the debugger is trying to use the system bus to write an instruction to implement the break point but fails because the system bus method is not supported and as such fails.

I also wanted to note that if you run openocd in step 2 instead of ash-riscv-gdb-server it also can cause Memory access -> System bus to show up in step 6. It doesn't happen always but its pretty common. Also note that openocd will use Program buffer to write memory over system bus and abstract access memory which is why is might be working while ash-riscv-gdb-server is failing (I'm assuming that it prioritizes System bus for memory access). Remember, if I use openocd in step 2 and step 6 then it all works.

Some other things I have tried to fix. Nothing worked:

1. Slowed the clock on my design down.
2. Slowed the JTAG clock down to 1MHZ.
3. Tried on both Nios V/g  and Nios V/m with the same results. I am using the Nios V/m in my design.

[XleraSolutions]

Hi all,

I see this same issue, for about 6mo now. I had thought it was because I was running the Risc-V from external DDR3. However, I eventually built a reduced case that had nothing but the minimum required for Risc-V, running from internal blockram. Still the issue persists. Only way to get it to work is reload the SOF.

A similar issue it that the processor traps on a bad event when loading the elf and running. Often just hitting restart several times will get it to make it to main().

Nios-II never exhibited this behavior. A working theory is that the Risc-V core is executing garbage or still executing your elf from the last run. I've noticed that stopping or disconnecting seems to release the processor to run from there (I don't think the Nios-II did that). In any event, something seems to not get reset correctly in the Risc-V core when reloading.

[BrendenB]

I just wanted to mention a work around I have been using for debugging. I have been using OpenOCD running externally from the RiscFree IDE.

1. Terminal: openocd-cfg-gen debugger.cfg
2. Terminal: openocd -f debugger.cfg
3. IDE: Create "GDB OpenOCD Debugging Launcher"
4. IDE: On Main Configure executable.
5. IDE: Setup to compile the program before launch.
6. IDE: On Debugger tab Uncheck "Start OpenOCD locally". I'm running it externally to make things easier. I guess you can launch it from here if you wanted.
7. IDE: Executable name ${eclipse_home}/../toolchain/riscv32-unknown-elf/bin/riscv32-unknown-elf-gdb. This was automatically filled.
8. IDE: Commands: set mem inaccessible-by-default off. Again automatically filled.
9. IDE: Remote Target Host: localhost Port: 3333. Again automatically filled.
10. IDE: I left the other settings at their defaults.

This seems to be pretty reliable to me (but slower).

[JingyangTeh_Altera]

 

Hi

I mean removing the connection in the debug tab.

You could right click on the connection and removing the connection.

(Please ignore the connection name in the screenshot below)

Have you tried using the setting from the how-to in the below link?

https://www.intel.com/content/www/us/en/docs/programmable/730783/23-4/setting-debug-configurations-and-downloading-59901.html

Regards

Jingyang, Teh

[JingyangTeh_Altera]

Hi

Any update on this case?

Regards

Jingyang, Teh

[BrendenB]

I deleted the connection and tried again. Same result. Works the first time. Wont work again until I re-program the SOF.

I also tried deleting the connection after the error occurs and creating a new one. The connection can never be re-established until I re-program the SOF.

As I mentioned above, I am able to reproduce this outside of the IDE just using the command line tools manually (running them the same way the IDE does in the background). I'm not sure the IDE is playing a part here. Seems to be something wrong with the ash-riscv-gdb-server and/or the IP itself.

[XleraSolutions]

I agree, I have been dealing with the same behavior to the point that reloading the SOF is a habit. Posing a problem now that development is further along (plus QSPI booting now), since reloading the SOF is a full reset to everything.

Thought the issue was related to using off chip DDR, but reduced the case to minimal nios-v running from block ram with only the essentials.

Nios-II didn’t have this issue.

[JingyangTeh_Altera]

Hi

Could you try enabling the "Issue TAP_Reset" option in the debugger configuration?

Regards

Jingyang, Teh

[khtan]

Hi,

May I know the status of this case, whether anyone tried on the enabling the "Issue TAP_Reset" option in the debugger configuration?

Thanks

Regards

Kian

[BrendenB]

I tried the option "Issue TAP_Reset" and still the same outcome. Works on the first connection. Wont work after until I re-program the SOF.

Is there some form of debugging that I can enable that might debug this problem?

[XleraSolutions]

same, the Issue TAP Reset does not affect the problem.

[JingyangTeh_Altera]

HI

After asking around, it seems like it is a tool issue and it have been fix in the latest quartus version.

Could you try migrating your design to a newer quartus version maybe 24.1pro ?

Regards

Jingyang, Teh

[BrendenB]

I can't go to a pro version because I am using a Cyclone V and its only supported on standard/lite versions.

However, I did download the 24.2 pro version is tried the newer riscfree in that version. It produced the same results. The first debug after SOF programming works fine. The second and subsequent debug fail until the SOF is re-programmed. The attached file shows the commands I ran to test. It broke both when using riscfree UI and the command line directly.

I had to use the SOF generated on standard as pro can't generate a SOF for my Cyclone V.

At what level was the bug fixed? Was it in the IP or the software? If its in the IP, I wont be able to check because its not possible to use pro to generate a SOF for my FPGA. If its in the software, then it still seems to be happening in 24.2 pro.

[JingyangTeh_Altera]

Hi

You will need to upgrade the whole project of the NiosV project.

Did you meet any error when upgrading the project to 24.2 pro?

It is an bug in rtl code of the NiosV core that is not compatible with tool.

Regards

Jingyang, Teh

[JingyangTeh_Altera]

Hi

Any update on this case?

Are you able to migrate the design and test out if the problem still occur?

Regards

Jingyang, Teh

[BrendenB]

I am using a Cyclone V FPGA which means I can't use pro as its not supported. See https://www.intel.com/content/www/us/en/support/programmable/support-resources/design-software/devices-support.html#cyclone%C2%AEseries

[JingyangTeh_Altera · Accepted solution]

Hi

Sorry in that cause you will need to wait for the release of Quartus Standard 24.1 in Q1 next year.

Regards

Jingyang, Teh

[BrendenB]

This was indeed fixed in Standard/Lite 24.1.

[JingyangTeh_Altera]

Hi

I’m glad that your question has been addressed, I now transition this thread to community support. If you have a new question, Please login to ‘https://supporttickets.intel.com/s/?language=en_US’, view details of the desire request, and post a feed/response within the next 15 days to allow me to continue to support you. After 15 days, this thread will be transitioned to community support. The community users will be able to help you on your follow-up questions.

Regards

Jingyang, Teh