topic Intel MPI debugging and profiling tools? in Intel® MPI Library

Intel MPI debugging and profiling tools?

j0e — Wed, 16 Nov 2011 17:08:04 GMT

Hi All,

I'm just getting started debugging and profiling MPI fortran code using the latest Intel Cluster Studio. For development, I largely use VS2010 and run under Windows on a single node (w/ 8 cpus) (production runs are on a linux cluster).

For basic debugging, I start an MPI run as usual, then just attach the debugger to all the processes via VS. Seems to work fine (at least at the basic level I'm running).

I've used the Intel Trace Collector/Analyzer, after compiling with /Qtcollect and running itcpin via mpiexec, such as: mpiexec -n 4 itcpin --insert VT --run -- myprogram.exe arguments This also seems to work as expected.

My only question, is this pretty much it?, or are there other Intel MPI tools that are useful that I'm not aware of?

Thanks for your help!
-joe

Intel MPI debugging and profiling tools?

Dmitry_K_Intel2 — Thu, 17 Nov 2011 10:13:15 GMT

Hi Joe,

I hope that you know about Message Checker. You just need to use 'VTmc' library instead of 'VT' for '--insert' opiton.

Also you can get Light-weght statistics be setting I_MPI_STATS=N (where N from 0 to 10) - you'll get text files with statistical information.
Debug information from the MPI library may be helpful sometimes - you just need to set I_MPI_DEBUG environment variable.

If you have Cluster Studio XE you can benefit from using of VTune Amplifier XE and Inspector XE applications.

BTW: if you compile your program with /Qtcollect you should not use itcpin - VT library has already been linked to the program. Just try to run 'mpiexec -n 4myprogram.exe'

Regards!
Dmitry

Intel MPI debugging and profiling tools?

j0e — Thu, 17 Nov 2011 13:38:31 GMT

Thanks Dmitry I'll give those suggestions a try. However, when I just use 'mpiexec -n 4myprogram.exe', the trace file myprogram.stf does not get created. I've run the program form the 'Intel Cluster Studio 2012 for Windows Build Environment', but that doesn't help. I've noticed the following:

If I don't compile with /Qtcollect, I get trace files when using itcpin, as expected
If I compile with /Qtcollect, but do not specify the VT.lib, I get link errors, so VT.lib is being linked to, but it does not produce trace file (at least not in the directory I would expect it to) if I do not run it with itcpin.
If I run itcpin with VTmc, I get the following errors at run time:

[bash][0] WARNING: EXCEPTION_ACCESS_VIOLATION occurred
[1] ERROR: Signal 3 caught in ITC code section.
[1] ERROR: Either ITC is faulty or (more likely in a release version)
[1] ERROR: the application has corrupted ITC's internal data structures.
[1] ERROR: Giving up now...
[0] ERROR: Signal 3 caught in ITC code section.
[0] ERROR: Either ITC is faulty or (more likely in a release version)
[0] ERROR: the application has corrupted ITC's internal data structures.
[0] ERROR: Giving up now...[/bash]

I seem to recall reading something about compiling/running MPI code if using static vs dynamic libraries, but I don't know if this has anything to do with that.

Also, I see reference to Intel Message Checker, and it sounds like there is a GUI for it, but I have not been able to find the software that reads the stf file for message checking.

cheers,
-joe