Debug user space application with MMU

[Altera_Forum]

I am trying to use the gdbserver as described here: 

http://nioswiki.com/linux/debug_user_space  At some point, this was working, but sometimes it wouldn't work and now I haven't had it working for a while. The application and gdb dies on a crash and doesn't let me debug. This isn't just this particular crash, I've had this happen with almost every crash while developing.  On the server:   --- Quote Start ---  root:/> gdbserver localhost:9999 myprogram Process myprogram created; pid = 50 Listening on port 9999 Remote debugging from host 192.168.100.28 <program output...>  Child terminated with signal = b  Child terminated with signal = 0xb (SIGSEGV) GDBserver exiting root:/>  --- Quote End ---    On the client:   --- Quote Start ---  Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists. (gdb) backtrace No stack.  --- Quote End ---    Any advice?  Also, the wiki says "we will add the gdbserver source to uclinux-dist later" -- is there any progress on this? Could the gdbserver currently in uclinux-dist work?

[Altera_Forum]

Debugging after a crash might be difficult. Are you sure it's just with NIOS ?  

 I just read a good book about working with gdb:   the art of debugging with gdb, ddd and eclipse: for professionals and students (http://www.amazon.de/art-debugging-gdb-eclipse-professionals/dp/1593271743/ref=sr_1_3?ie=utf8&s=books-intl-de&qid=1266605020&sr=8-3) by Norman Matloff und P. J. Salzma  -Michael

[Altera_Forum]

In the cases when I could run the program on my PC, I could debug the crash using gdb, so it is something with the NIOS gdbserver provided with the mmu toolchain.

[Altera_Forum]

Just wanted to add that breakpoints don't work either, with the same result, the program just terminates: 

Program terminated with signal SIGTRAP, Trace/breakpoint trap.  Not having the debugger available is really eating up a lot of time :-(

[Altera_Forum]

Hi, 

   --- Quote Start ---  Just wanted to add that breakpoints don't work either, with the same result, the program just terminates:   --- Quote End ---     When the debuggee program crashes, the 'gdbserver' will terminate itself. So please get debuggee's 'coredump' file by setting 'ulimit -c ***', for example in your 'bash', And copy it to your PC, and read it by Nios's 'gdb' with the command 'core-file core'. I think that you can reach the position which causes the crash. Then set a breakpoint there and...  And don't forget to compile your program with the option switch '-O0'.    --- Quote Start ---   Program terminated with signal SIGTRAP, Trace/breakpoint trap.   --- Quote End ---    SIGTRAP? Really?  Kazu

[Altera_Forum]

I realized when somebody mentioned it in the other thread on this topic (http://www.alteraforum.com/forum/showthread.php?t=19322) that I am not actually quite following the directions there. 

 I am not using set solib-absolute-prefix ~/nios2-linux/toolchain-mmu/x86-linux2/nios2-wrs-linux-gnu/libc but instead the libs in the romfs: set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/  Since all the libraries I am using need to be on the device for the application to run, I think this should not make a difference.

[Altera_Forum]

--- Quote Start ---  When the debuggee program crashes, the 'gdbserver' will terminate itself. So please get debuggee's 'coredump' file by setting 'ulimit -c ***', for example in your 'bash', And copy it to your PC, and read it by Nios's 'gdb' with the command 'core-file core'. I think that you can reach the position which causes the crash. Then set a breakpoint there and...  And don't forget to compile your program with the option switch '-O0'.  --- Quote End ---   I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is: warning: Couldn't find general-purpose registers in core file. # 0 0x00000000 in ?? ()

[Altera_Forum]

Hi, 

   --- Quote Start ---  I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is: warning: Couldn't find general-purpose registers in core file.# 0 0x00000000 in ?? ()  --- Quote End ---     Do you have enough space for the core dump file? For example,  ulimit -c unlimiled  Kazu

[Altera_Forum]

Hi, 

 I am debugging a user application and I have some doubts, and maby someone can answer my questions.  My application produces a segmentation fault, I suppose that a segmentation fault provoke a core dumped, is this true?  If I want to have core dumps in uClinux, as kazuyasu commented in last posts I have to insert the command : ulimit -c unlimited. But where is it in the configuration kernel?, I haven't found.  And my last question is, is the core file created in /proc/sys/kernel?, if not where is it found?  Many thanks

[Altera_Forum]

Hi, 

   --- Quote Start ---   My application produces a segmentation fault, I suppose that a segmentation fault provoke a core dumped, is this true?   --- Quote End ---     Yes, but the dumping is limited by 'core file size', and it's 0 by default.    --- Quote Start ---   I have to insert the command : ulimit -c unlimited. But where is it in the configuration kernel?, I haven't found.   --- Quote End ---     'ulimit -c ***' that I mentioned before is an internal command of 'bash' and you don't need to configure your kernel in general. See also  http://linux.die.net/man/3/ulimit     --- Quote Start ---    And my last question is, is the core file created in /proc/sys/kernel?, if not where is it found?   --- Quote End ---     In general, the core dump file is made in your 'current working directory'.  http://linux.die.net/man/5/core  Kazu

[Altera_Forum]

Hi Kazu, 

 I have changed the shell to ash and now I can see the core file.  Many thanks

[Altera_Forum]

This may be related to troubles debugging. I am trying to print a stack trace from within the program, but that doesn't work either. I tried with a very simple non-threaded C program: 

#include <stdio.h> # include <execinfo.h> # include <stdlib.h> void print_trace() { void *array; size_t size; // get void*'s for all entries on the stack size = backtrace(array, 10); // print out all the frames to stderr fprintf(stderr, "trace size %zu:\n", size); backtrace_symbols_fd(array, size, 2); exit(1); } void baz() { print_trace(); } void bar() { baz(); } void foo() { bar(); } int main(int argc, char **argv) { foo(); }   When I build this on my PC (Ubuntu 10.10 x86_64) I get a nice stacktrace: gcc -g -rdynamic -o test_bt test_bt.c trace size 7: ./test_bt(print_trace+0x19) ./test_bt(baz+0xe) ./test_bt(bar+0xe) ./test_bt(foo+0xe) ./test_bt(main+0x19) /lib/libc.so.6(__libc_start_main+0xfe) ./test_bt On Nios II Linux I just get this: nios2-linux-gnu-gcc -g -rdynamic -o test_bt test_bt.c trace size 1:   I've been trying to mess with compiler flags, but haven't found any magic so far. Any ideas?

[Altera_Forum]

--- Quote Start ---  I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is: warning: Couldn't find general-purpose registers in core file. # 0 0x00000000 in ?? ()  --- Quote End ---    I hit the same problem here too and traced it down to the kernel writing a faulty corefile, missing the .reg and .reg/<N> section. The cause for this was ELF_NGREG in arch/nios2/include/asm/elf.h being too small ((sizeof (struct pt_regs) / sizeof(elf_greg_t)), while ELF_CORE_COPY_REGS also writes all registers from struct switch_stack to the buffer.  If ELF_NGREG is defined as in the following patch, the corefile is generated properly and can be used with nios2-linux-gnu-gdb:   -#define ELF_NGREG (sizeof (struct pt_regs) / sizeof(elf_greg_t)) +#define ELF_NGREG ((sizeof(struct pt_regs) + sizeof(struct switch_stack)) / sizeof(elf_greg_t))   I'll also apply this fix to the nios2 branch of the nios2-linux kernel git repository.

[Altera_Forum]

Tobias, I just saw this fix in the commit log and was surprised to see a link back to my report here because I missed your response in the thread. I will try this fix!

[Altera_Forum]

--- Quote Start ---  This may be related to troubles debugging. I am trying to print a stack trace from within the program, but that doesn't work either.  --- Quote End ---    Have you get it to work, ykozlov? I couldn't print out a stacktrace in a user-level program with backtrace() / backtrace_symbols()...

[Altera_Forum]

--- Quote Start ---  Have you get it to work, ykozlov? I couldn't print out a stacktrace in a user-level program with backtrace() / backtrace_symbols()...  --- Quote End ---   No, I never got that working, but tklauser's kernel fix did fix debugging using a core dump.

[Altera_Forum]

--- Quote Start ---  No, I never got that working, but tklauser's kernel fix did fix debugging using a core dump.  --- Quote End ---    And I fixed the backtrace() function... or bypass it and wrote my own!  stackframe.c # define _GNU_SOURCE # include <stdio.h># include <inttypes.h># include <stdint.h># include <ucontext.h># include <dlfcn.h># include <string.h> struct frame { void *fp; // frame pointer void *ra; // return address }; void print_symbol(void *address) { Dl_info info; char library; char function; void *symbol_address; if (dladdr(address, &info) == 0) { strncpy(function, "???", sizeof(function)); strncpy(library, "???", sizeof(library)); symbol_address = 0; } else { strncpy(function, info.dli_sname, sizeof(function)); strncpy(library, info.dli_fname, sizeof(library)); symbol_address = info.dli_saddr; } printf("0x%08X %s: %s\n", (uint32_t) symbol_address, library, function); } void backtrace() { struct frame *frame; // frame = CURRENT_FRAME_POINTER __asm("mov %0,r28" : "=r"(frame)); // Last frames while (frame && frame->fp) { printf("fp = 0x%08X ra = 0x%08X\n", (uint32_t) frame->fp, (uint32_t) frame->ra); print_symbol((void *) frame->ra); frame = (struct frame *) frame->fp; } } void test3() { backtrace(); } void test2() { test3(); } void test1() { test2(); } int main(int argc, char *argv) { test1(); return 0; }Makefile CC = nios2-linux-gnu-gcc CFLAGS = -Wall -fno-omit-frame-pointer LDFLAGS = -rdynamic -ldl SOURCES = stackframe.c OBJECTS = $(SOURCES:%.c=%.o) PROGRAM = stackframe .PHONY: all clean all: $(PROGRAM) clean: rm -rf $(OBJECTS) $(PROGRAM) build: clean all $(PROGRAM): $(OBJECTS) $(CC) -o $@ $(OBJECTS) $(LDFLAGS) %.o: %.c $(CC) $(CFLAGS) -c $< Build: $ make nios2-linux-gnu-gcc -Wall -fno-omit-frame-pointer -c stackframe.c nios2-linux-gnu-gcc -o stackframe stackframe.o -rdynamic -ldl Execute: # stackframe fp = 0x7FE81D24 ra = 0x00001838 0x00001824 stackframe: test3 fp = 0x7FE81D2C ra = 0x00001860 0x0000184C stackframe: test2 fp = 0x7FE81D34 ra = 0x00001888 0x00001874 stackframe: test1 fp = 0x7FE81D44 ra = 0x000018B8 0x0000189C stackframe: main