Serial I/O runtime keeps crashing!

[Altera_Forum]

Hello, I have a Nios II system with two UARTs, both of which have worked perfectly fine for a while... Then I made a couple of unrelated changes to my Quartus design (outside the Nios system), and ever since then, my Nios code has been crashing (hanging or restarting) somewhere in the serial I/O libraries or system runtime, at unpredictable times, but usually within the first 10 or 100 calls to serial I/O library functions (fprintf(), fgets(), etc.) 

 Unfortunately, I cannot pinpoint the cause of the problem in the Nios II IDE debugger, because it doesn't seem to include debuggable versions of the system runtime - I can step through the outer layers of the STDIO libraries, but not e.g. into read(), which is one of the places where it's crashing.  What I was wondering is the following:  1) Is it possible that changes to the outside design could shift placement/routing in ways that cause subtle timing glitches within the Avalon UART core IP to manifest themselves? If so, it's disappointing that the IP is so fragile... and I'm not sure how to fix it, except by backing off to the last known working version of the Quartus design...  2) How does one usually debug or workaround these kinds of problems? It's very frustrating, because the code crashes in a different place each time, and I can't even step into the point where it's crashing with the source-level debugger. I don't have source code for the system runtime or the megafunction IP. (Or is there some way to get it?)  Thanks for any advice, -Mike

[Altera_Forum]

--- Quote Start ---   1) Is it possible that changes to the outside design could shift placement/routing in ways that cause subtle timing glitches within the Avalon UART core IP to manifest themselves?    --- Quote End ---   Yes, it is possible. Is your design correctly timing constrained? And does it meet timing after compilation?    --- Quote Start ---   2) How does one usually debug or workaround these kinds of problems? It's very frustrating, because the code crashes in a different place each time, and I can't even step into the point where it's crashing with the source-level debugger. I don't have source code for the system runtime or the megafunction IP.   --- Quote End ---   I think you should be able to step into the read(),fgets()... functions at the assembler level, even if you don't have source code, in order to identify the exact point where your code hangs.  Maybe you can not debug into those functions just because the hang/restart behavior happens exactly when you call the function. Then you possibly have a system stack overflow. Do you use big local variables or recursive functions?

[Altera_Forum]

@Cris72: Thanks for the reply. I don't know how to place appropriate timing constraints on the Nios SOPC system, since I didn't design the IP cores used there. I would have expected that any needed timing constraints would have been generated automatically together with the modules generated by SOPC Builder. Possibly if I had source code for the UART core IP, I could improve on the timing constraints, or otherwise debug the IP, but I don't know if that's available. As it is, I didn't notice any timing errors during the Quartus workflow.  

 The problem with trying to debug the assembler code is knowing where to focus attention - the crashes seem to be happening nondeterministically, at seemingly random time points; and when the crash happens, I might find myself in an endless loop, or back in the system initialization code, but I don't know how/why I got there.   And when single-stepping, the behavior is different yet again, because the external systems that are generating our serial input asynchronously aren't being paused by the Nios debugger. So stepping through the code ends up not being very helpful.  I don't think it's a stack problem... My code isn't recursive and has only a few levels of functions. But I'll try cutting down on the local variables and see if that helps...

[Altera_Forum]

First of all I'd suggest you connect your serial input to a "controllable" source, I mean any serial port which lets you decide when and what it transmits. 

Although I don't know if this is feasible in your system, the straightforward setup would be connecting your UART to a PC serial port and using a Hyperterminal-like program to debug serial activity.

[Altera_Forum]

Don't use hyperterm, it will confuse you further. 

putty has a much better async terminal emulater.

[Altera_Forum]

Thanks to all who replied... The first poster's remark about stack overflow reminded me to go back and double-check the stack size... Turns out that after linking the latest version of my firmware to the 64K RAM I had defined, I had only about 1,800 bytes left for stack + heap! Even without any recursion, no big objects on the stack, and nothing except maybe a handful of serial buffers and FD structures allocated by the system runtime on the heap, 1.5K RAM sounded pretty tight... So, I doubled my RAM size from 64K to 128K in SOPC Builder, and this seems to have completely solved the problem!  

 So, the stack must have just been stomping over some heap items, or vice-versa. This accounts for the weird behavior I saw, with nondeterministic crashes & occasional garbled output. The unrelated features that I had added to the code since the last time it was working must have grown the program text just large enough so that inadequate space then remained for the run-time memory requirements.   Thanks again for everyone's help! I'm glad it wasn't an IP problem.

[Altera_Forum]

Stack crashing into heap is very confusing, been there fixed that :-)