Nios® V/II Embedded Design Suite (EDS)
Support for Embedded Development Tools, Processors (SoCs and Nios® V/II processor), Embedded Development Suites (EDSs), Boot and Configuration, Operating Systems, C and C++
12620 Discussions

Debug user space application with MMU

Altera_Forum
Honored Contributor II
2,473 Views

I am trying to use the gdbserver as described here: 

http://nioswiki.com/linux/debug_user_space 

 

At some point, this was working, but sometimes it wouldn't work and now I haven't had it working for a while. The application and gdb dies on a crash and doesn't let me debug. This isn't just this particular crash, I've had this happen with almost every crash while developing. 

 

On the server: 

 

--- Quote Start ---  

root:/> gdbserver localhost:9999 myprogram 

Process myprogram created; pid = 50 

Listening on port 9999 

Remote debugging from host 192.168.100.28 

<program output...> 

 

Child terminated with signal = b 

 

Child terminated with signal = 0xb (SIGSEGV) 

GDBserver exiting 

root:/> 

--- Quote End ---  

 

 

On the client: 

 

--- Quote Start ---  

Program terminated with signal SIGSEGV, Segmentation fault. 

The program no longer exists. 

(gdb) backtrace 

No stack. 

--- Quote End ---  

 

 

Any advice? 

 

Also, the wiki says "we will add the gdbserver source to uclinux-dist later" -- is there any progress on this? Could the gdbserver currently in uclinux-dist work?
0 Kudos
16 Replies
Altera_Forum
Honored Contributor II
1,282 Views

Debugging after a crash might be difficult. Are you sure it's just with NIOS ?  

 

I just read a good book about working with gdb:  

the art of debugging with gdb, ddd and eclipse: for professionals and students (http://www.amazon.de/art-debugging-gdb-eclipse-professionals/dp/1593271743/ref=sr_1_3?ie=utf8&s=books-intl-de&qid=1266605020&sr=8-3) by Norman Matloff und P. J. Salzma 

 

-Michael
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

In the cases when I could run the program on my PC, I could debug the crash using gdb, so it is something with the NIOS gdbserver provided with the mmu toolchain.

0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Just wanted to add that breakpoints don't work either, with the same result, the program just terminates: 

Program terminated with signal SIGTRAP, Trace/breakpoint trap. 

 

Not having the debugger available is really eating up a lot of time :-(
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Hi, 

 

 

--- Quote Start ---  

Just wanted to add that breakpoints don't work either, with the same result, the program just terminates: 

 

--- Quote End ---  

 

 

When the debuggee program crashes, the 'gdbserver' will terminate itself. So please get debuggee's 'coredump' file by setting 'ulimit -c ***', for example in your 'bash', And copy it to your PC, and read it by Nios's 'gdb' with the command 'core-file core'. I think that you can reach the position which causes the crash. Then set a breakpoint there and... 

And don't forget to compile your program with the option switch '-O0'. 

 

 

--- Quote Start ---  

 

Program terminated with signal SIGTRAP, Trace/breakpoint trap. 

 

--- Quote End ---  

 

 

SIGTRAP? Really? 

 

Kazu
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

I realized when somebody mentioned it in the other thread on this topic (http://www.alteraforum.com/forum/showthread.php?t=19322) that I am not actually quite following the directions there. 

 

I am not using 

set solib-absolute-prefix ~/nios2-linux/toolchain-mmu/x86-linux2/nios2-wrs-linux-gnu/libc 

but instead the libs in the romfs: 

set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ 

 

Since all the libraries I am using need to be on the device for the application to run, I think this should not make a difference.
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

 

--- Quote Start ---  

When the debuggee program crashes, the 'gdbserver' will terminate itself. So please get debuggee's 'coredump' file by setting 'ulimit -c ***', for example in your 'bash', And copy it to your PC, and read it by Nios's 'gdb' with the command 'core-file core'. I think that you can reach the position which causes the crash. Then set a breakpoint there and... 

And don't forget to compile your program with the option switch '-O0'. 

--- Quote End ---  

 

I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is: 

warning: Couldn't find general-purpose registers in core file. # 0 0x00000000 in ?? ()
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Hi, 

 

 

--- Quote Start ---  

I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is: 

warning: Couldn't find general-purpose registers in core file.# 0 0x00000000 in ?? () 

--- Quote End ---  

 

 

Do you have enough space for the core dump file? For example, 

 

ulimit -c unlimiled 

 

Kazu
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Hi, 

 

I am debugging a user application and I have some doubts, and maby someone can answer my questions. 

 

My application produces a segmentation fault, I suppose that a segmentation fault provoke a core dumped, is this true? 

 

If I want to have core dumps in uClinux, as kazuyasu commented in last posts I have to insert the command : ulimit -c unlimited. But where is it in the configuration kernel?, I haven't found. 

 

And my last question is, is the core file created in /proc/sys/kernel?, if not where is it found? 

 

Many thanks
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Hi, 

 

 

--- Quote Start ---  

 

My application produces a segmentation fault, I suppose that a segmentation fault provoke a core dumped, is this true? 

 

--- Quote End ---  

 

 

Yes, but the dumping is limited by 'core file size', and it's 0 by default. 

 

 

--- Quote Start ---  

 

I have to insert the command : ulimit -c unlimited. But where is it in the configuration kernel?, I haven't found. 

 

--- Quote End ---  

 

 

'ulimit -c ***' that I mentioned before is an internal command of 'bash' and you don't need to configure your kernel in general. See also 

 

http://linux.die.net/man/3/ulimit 

 

 

--- Quote Start ---  

 

And my last question is, is the core file created in /proc/sys/kernel?, if not where is it found? 

 

--- Quote End ---  

 

 

In general, the core dump file is made in your 'current working directory'. 

 

http://linux.die.net/man/5/core 

 

Kazu
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Hi Kazu, 

 

I have changed the shell to ash and now I can see the core file. 

 

Many thanks
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

This may be related to troubles debugging. I am trying to print a stack trace from within the program, but that doesn't work either. I tried with a very simple non-threaded C program: 

#include <stdio.h> # include <execinfo.h> # include <stdlib.h> void print_trace() { void *array; size_t size; // get void*'s for all entries on the stack size = backtrace(array, 10); // print out all the frames to stderr fprintf(stderr, "trace size %zu:\n", size); backtrace_symbols_fd(array, size, 2); exit(1); } void baz() { print_trace(); } void bar() { baz(); } void foo() { bar(); } int main(int argc, char **argv) { foo(); }  

 

When I build this on my PC (Ubuntu 10.10 x86_64) I get a nice stacktrace: 

gcc -g -rdynamic -o test_bt test_bt.c 

trace size 7: ./test_bt(print_trace+0x19) ./test_bt(baz+0xe) ./test_bt(bar+0xe) ./test_bt(foo+0xe) ./test_bt(main+0x19) /lib/libc.so.6(__libc_start_main+0xfe) ./test_bt 

On Nios II Linux I just get this: 

nios2-linux-gnu-gcc -g -rdynamic -o test_bt test_bt.c 

trace size 1:  

 

I've been trying to mess with compiler flags, but haven't found any magic so far. Any ideas?
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

 

--- Quote Start ---  

I am trying to use a core file now. I am running `nios2-linux-gnu-gdb mynios2program`, with set solib-absolute-prefix /data/nios/nios2-linux/uClinux-dist/romfs/ in .gdbinit. Then core-file mycorefile, but all I can get is: 

warning: Couldn't find general-purpose registers in core file. # 0 0x00000000 in ?? () 

--- Quote End ---  

 

 

I hit the same problem here too and traced it down to the kernel writing a faulty corefile, missing the .reg and .reg/<N> section. The cause for this was ELF_NGREG in arch/nios2/include/asm/elf.h being too small ((sizeof (struct pt_regs) / sizeof(elf_greg_t)), while ELF_CORE_COPY_REGS also writes all registers from struct switch_stack to the buffer. 

 

If ELF_NGREG is defined as in the following patch, the corefile is generated properly and can be used with nios2-linux-gnu-gdb: 

 

-#define ELF_NGREG (sizeof (struct pt_regs) / sizeof(elf_greg_t)) +#define ELF_NGREG ((sizeof(struct pt_regs) + sizeof(struct switch_stack)) / sizeof(elf_greg_t))  

 

I'll also apply this fix to the nios2 branch of the nios2-linux kernel git repository.
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

Tobias, I just saw this fix in the commit log and was surprised to see a link back to my report here because I missed your response in the thread. I will try this fix!

0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

 

--- Quote Start ---  

This may be related to troubles debugging. I am trying to print a stack trace from within the program, but that doesn't work either. 

--- Quote End ---  

 

 

Have you get it to work, ykozlov? I couldn't print out a stacktrace in a user-level program with backtrace() / backtrace_symbols()...
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

 

--- Quote Start ---  

Have you get it to work, ykozlov? I couldn't print out a stacktrace in a user-level program with backtrace() / backtrace_symbols()... 

--- Quote End ---  

 

No, I never got that working, but tklauser's kernel fix did fix debugging using a core dump.
0 Kudos
Altera_Forum
Honored Contributor II
1,282 Views

 

--- Quote Start ---  

No, I never got that working, but tklauser's kernel fix did fix debugging using a core dump. 

--- Quote End ---  

 

 

And I fixed the backtrace() function... or bypass it and wrote my own! 

 

stackframe.c # define _GNU_SOURCE # include <stdio.h># include <inttypes.h># include <stdint.h># include <ucontext.h># include <dlfcn.h># include <string.h> struct frame { void *fp; // frame pointer void *ra; // return address }; void print_symbol(void *address) { Dl_info info; char library; char function; void *symbol_address; if (dladdr(address, &info) == 0) { strncpy(function, "???", sizeof(function)); strncpy(library, "???", sizeof(library)); symbol_address = 0; } else { strncpy(function, info.dli_sname, sizeof(function)); strncpy(library, info.dli_fname, sizeof(library)); symbol_address = info.dli_saddr; } printf("0x%08X %s: %s\n", (uint32_t) symbol_address, library, function); } void backtrace() { struct frame *frame; // frame = CURRENT_FRAME_POINTER __asm("mov %0,r28" : "=r"(frame)); // Last frames while (frame && frame->fp) { printf("fp = 0x%08X ra = 0x%08X\n", (uint32_t) frame->fp, (uint32_t) frame->ra); print_symbol((void *) frame->ra); frame = (struct frame *) frame->fp; } } void test3() { backtrace(); } void test2() { test3(); } void test1() { test2(); } int main(int argc, char *argv) { test1(); return 0; }Makefile CC = nios2-linux-gnu-gcc CFLAGS = -Wall -fno-omit-frame-pointer LDFLAGS = -rdynamic -ldl SOURCES = stackframe.c OBJECTS = $(SOURCES:%.c=%.o) PROGRAM = stackframe .PHONY: all clean all: $(PROGRAM) clean: rm -rf $(OBJECTS) $(PROGRAM) build: clean all $(PROGRAM): $(OBJECTS) $(CC) -o $@ $(OBJECTS) $(LDFLAGS) %.o: %.c $(CC) $(CFLAGS) -c $< Build: $ make nios2-linux-gnu-gcc -Wall -fno-omit-frame-pointer -c stackframe.c nios2-linux-gnu-gcc -o stackframe stackframe.o -rdynamic -ldl Execute: # stackframe fp = 0x7FE81D24 ra = 0x00001838 0x00001824 stackframe: test3 fp = 0x7FE81D2C ra = 0x00001860 0x0000184C stackframe: test2 fp = 0x7FE81D34 ra = 0x00001888 0x00001874 stackframe: test1 fp = 0x7FE81D44 ra = 0x000018B8 0x0000189C stackframe: main
0 Kudos
Reply