How to locate an error in a parallel program.

[ys2009]

Recently I noticed that my code will crash due to unknown reasons when I ran some certain cases and this happens only after it has been running successfully for hundreds of time steps. There must be a bug somewhere within the code but the problem is I have no way to locate it. The error message is something like this:

Image PC Routine Line Source
mca_btl_openib.so 00002AAAB12BB38D Unknown Unknown Unknown
mca_bml_r2.so 00002AAAB10B13DD Unknown Unknown Unknown
libopen-pal.so.0 00002AAAAB43E1A0 Unknown Unknown Unknown
libmpi.so.0 00002AAAAAF2350C Unknown Unknown Unknown
libmpi.so.0 00002AAAAAF54F7C Unknown Unknown Unknown
libmpi_f77.so.0 00002AAAAACDA9DC Unknown Unknown Unknown
tetrin_pmg 00000000004D0DE2 Unknown Unknown Unknown
tetrin_pmg 000000000049D8BC Unknown Unknown Unknown
tetrin_pmg 0000000000418A52 Unknown Unknown Unknown
tetrin_pmg 0000000000415A5C Unknown Unknown Unknown
libc.so.6 0000003A8081D8B4 Unknown Unknown Unknown
tetrin_pmg 0000000000415969 Unknown Unknown Unknown

What specifically do I need to do with my code to let both the line and Source columns contain more useful info other than just unknown, they make it impossible for me to locate the bugs in the code.

the modules I've loaded include:

1) intel/compiler/64/11.0/074 2) openmpi/intel/1.2.8 3) sge/6.2

Thanks a lot!

[Ron_Green]

Quoting - ys2009

Recently I noticed that my code will crash due to unknown reasons when I ran some certain cases and this happens only after it has been running successfully for hundreds of time steps. There must be a bug somewhere within the code but the problem is I have no way to locate it. The error message is something like this:

Image PC Routine Line Source
mca_btl_openib.so 00002AAAB12BB38D Unknown Unknown Unknown
mca_bml_r2.so 00002AAAB10B13DD Unknown Unknown Unknown
libopen-pal.so.0 00002AAAAB43E1A0 Unknown Unknown Unknown
libmpi.so.0 00002AAAAAF2350C Unknown Unknown Unknown
libmpi.so.0 00002AAAAAF54F7C Unknown Unknown Unknown
libmpi_f77.so.0 00002AAAAACDA9DC Unknown Unknown Unknown
tetrin_pmg 00000000004D0DE2 Unknown Unknown Unknown
tetrin_pmg 000000000049D8BC Unknown Unknown Unknown
tetrin_pmg 0000000000418A52 Unknown Unknown Unknown
tetrin_pmg 0000000000415A5C Unknown Unknown Unknown
libc.so.6 0000003A8081D8B4 Unknown Unknown Unknown
tetrin_pmg 0000000000415969 Unknown Unknown Unknown

What specifically do I need to do with my code to let both the line and Source columns contain more useful info other than just unknown, they make it impossible for me to locate the bugs in the code.

the modules I've loaded include:

1) intel/compiler/64/11.0/074 2) openmpi/intel/1.2.8 3) sge/6.2

Thanks a lot!

You will need to compile and link with -g -traceback. You can still use other optimizations but make sure you call out -O2 or -O3 as -g will set to -O0 unless you explicitly call out a -O level.