Intel Fortran debugger and MPICH

[truedas]

Hi,
I want to debug this parallel Fortran program that I'm trying to make run on a Linux-type cluster using the Intel Fortran compiler (v.9.1.045) and debugger (v.9.1-28) and MPICH2.1.2. While building the executable with the -g option is straightforward, but when trying to invoke the compiler it crashes with this message:
$ idb -parallel mpiexec -machinefile machines -n 4 ./stagyympi
Intel Debugger for applications running on IA-32, Version 9.1-28, Build 20070305
execve failed: No such file or directory
Error: could not start debuggee
Could not start process for mpiexec
No image loaded ... Recovering ...

Any ideas what may be wrong here?
Thomas

[TimP]

There was a thread several years ago about idb (now named idbc) and mpich:
http://software.intel.com/en-us/forums/showthread.php?t=50526
Is there nothing about updates to this on the mpich2 support sites? Did it change that much?
Your command line appears to invoke debugging of mpiexec, not of your program.

[truedas]

Quoting - tim18

There was a thread several years ago about idb (now named idbc) and mpich:
http://software.intel.com/en-us/forums/showthread.php?t=50526
Is there nothing about updates to this on the mpich2 support sites? Did it change that much?
Your command line appears to invoke debugging of mpiexec, not of your program.

I guess there have been some changes. I have tried out what was suggested in that thread and it didn't work either:
neither mpirun nor the now recommended mpiexec recognize -dbg=idb as a valid option, whereas invoking it as
mpiexec -machinefile machines -n 1 idb stagyympi
yields:
problem with execution of idb on compute-0-0.local: [Errno 2] No such file or directory
So, no luck there. Any other ideas?
Thomas

[Dmitry_K_Intel2]

Quoting - truedas

Hi,

$ idb -parallel mpiexec -machinefile machines -n 4 ./stagyympi

Any ideas what may be wrong here?
Thomas

Hi Thomas,

Thank you for the interest to Intel products.

The command line looks very strange. It should look like:
"xterm -e idb -pid -mpi2 -parallel /usr/bin/python"

Also you should set environment variables:
IDB_HOME=
LD_LIBRARY_PATH=
IDB_PARALLEL_SHELL= - you don't need this var if you use rsh connection.
Probably, MPIEXEC_DEBUG=1

I hope this helps.

Best wishes,
Dmitry

[truedas]

Quoting - Dmitry Kuzmin (Intel)

Hi Thomas,

Thank you for the interest to Intel products.

The command line looks very strange. It should look like:
"xterm -e idb -pid -mpi2 -parallel /usr/bin/python"

Also you should set environment variables:
IDB_HOME=
LD_LIBRARY_PATH=
IDB_PARALLEL_SHELL= - you don't need this var if you use rsh connection.
Probably, MPIEXEC_DEBUG=1

I hope this helps.

Best wishes,
Dmitry

Hi Dmitry,
I finally got to trying that out, but setting the IDB_PARALLEL_SHELL and MPIEXEC_DEBUG as you said made things hang. When I have only IDB_HOME and LD_LIBRARY_PATH set, I can somehow start the debugger and it seems to try to attach somehow to the process (although I don't understand why the last argument is the python executable and not the actual program executable), but then it says:
Intel Debugger for applications running on IA-32, Version 9.1-28, Build 20070305
Reading symbolic information from /opt/rocks/bin/python2.4...No debugging symbols found
Attached to process id 27609 ....
stopped at [ _dl_sysinfo_int80(...) 0x008ca7a2]
Symbol "MPIR_being_debugged" is not defined.
Symbol "MPIR_Breakpoint" is not defined.
Symbol "MPIR_Breakpoint" is not defined.
::MPIR_Breakpoint has no valid breakpoint address
Make breakpoint pending on future shared library load? (y or )
Warning: Breakpoint not set

and hangs, no matter whether I have chosen y or n in the breakpoint question.

So, as that failed, I was prompted to look back into my MPICH installation again and notice that for some reason, it says in the configure log:
configure: WARNING: unrecognized options: --enable-debuginfo

although according to the installation guide, this is the option I should give to "provide access to the message queues for debuggers". In the configure log, it also says later:
checking debugging support... no
although I'm not sure whether that refers to this kind of debugging or rather to the debugging options for MPICH2 developpers themselves (which would not be of interest for me).
So basically, I'm rather confused about the error messages and wonder if I have to build MPICH2 in a different way to make the Intel debugger work (and if so, how).
May the problem lie here already, and how do you build MPICH2 then?
Thomas

[Dmitry_K_Intel2]

Quoting - truedas

Hi Dmitry,
I finally got to trying that out, but setting the IDB_PARALLEL_SHELL and MPIEXEC_DEBUG as you said made things hang. When I have only IDB_HOME and LD_LIBRARY_PATH set, I can somehow start the debugger and it seems to try to attach somehow to the process (although I don't understand why the last argument is the python executable and not the actual program executable), but then it says:
Intel Debugger for applications running on IA-32, Version 9.1-28, Build 20070305
Reading symbolic information from /opt/rocks/bin/python2.4...No debugging symbols found
Attached to process id 27609 ....
stopped at [ _dl_sysinfo_int80(...) 0x008ca7a2]
Symbol "MPIR_being_debugged" is not defined.
Symbol "MPIR_Breakpoint" is not defined.
Symbol "MPIR_Breakpoint" is not defined.
::MPIR_Breakpoint has no valid breakpoint address
Make breakpoint pending on future shared library load? (y or )
Warning: Breakpoint not set

and hangs, no matter whether I have chosen y or n in the breakpoint question.

So, as that failed, I was prompted to look back into my MPICH installation again and notice that for some reason, it says in the configure log:
configure: WARNING: unrecognized options: --enable-debuginfo

although according to the installation guide, this is the option I should give to "provide access to the message queues for debuggers". In the configure log, it also says later:
checking debugging support... no
although I'm not sure whether that refers to this kind of debugging or rather to the debugging options for MPICH2 developpers themselves (which would not be of interest for me).
So basically, I'm rather confused about the error messages and wonder if I have to build MPICH2 in a different way to make the Intel debugger work (and if so, how).
May the problem lie here already, and how do you build MPICH2 then?
Thomas

Hi Thomas,

I was talking about Intel MPI Library and IDB.
I don't know how MPICH supports IDB, but invocation looked incorrectly. Probably MPICH was configured incorrecly during installation.
An application hangs because of MPIEXEC_DEBUG variable.

IDB attaches to a python script first to get information about running processes from mtv.so. Mpiexec should wait for a debugger.

BTW: your IDB is very old... March 2007 - I'm not sure that IDB could work correctly with MPI-2 applications at that time. Could you get idb 10.1 somehow?

Regards!
Dmitry

[Dmitry_K_Intel2]

Some clarifications about idb 9.1 usage with mpich:

The correct start up command is:
mpirun -dbg=idb -np

Two pre-requisites:
1. The environment variable IDB_HOME needs to be set to the directory where IDB resides.
2. The file mpirun_dbg.idb, which is included in the IDB 9.1 distribution, needs to be put in the "bin" directory of the MPICH installation.

Please try this out and let me know if this helps.

Best wishes,
Dmitry

[truedas]

Quoting - Dmitry Kuzmin (Intel)

Some clarifications about idb 9.1 usage with mpich:

The correct start up command is:
mpirun -dbg=idb -np

Two pre-requisites:
1. The environment variable IDB_HOME needs to be set to the directory where IDB resides.
2. The file mpirun_dbg.idb, which is included in the IDB 9.1 distribution, needs to be put in the "bin" directory of the MPICH installation.

Please try this out and let me know if this helps.

Best wishes,
Dmitry

Hi Dmitry,
thanks for the info, and sorry for being slow with the response.

I guess it will be difficult to get my hands on a more recent version of the compiler on that architecture, but I tried it now with the version, and it still doesn't work.
$ echo $IDB_HOME
/opt/intel/idb/9.1.045/
I also copied mpirun_dbg.idb into the bin directory where the MPICH executables live, but neither mpirun nor mpiexec recognize the option -gdb:
$ mpirun -gdb=idb -np 4 stagyympi
invalid "local" arg: -gdb=idb

I have looked in the installation guide if there is a way to "teach" MPICH the use of compilers other than gdb, but have found none.

Thomas

[Dmitry_K_Intel2]

Quoting - truedas

I have looked in the installation guide if there is a way to "teach" MPICH the use of compilers other than gdb, but have found none.

Thomas

I've looked into the MPICH2 mpiexec.py script and seems they support only GDB (-gdb) and TV (-tv) debuggers. -gdb option doesn't have parameters.
Unfortunately it's not so easy to modify mpiexec.py so that it would work with idb. If you wanna use IDB you probably need to install Intel MPI Library.

Regards!
Dmitry