Unable to generate trace file with mpirun

[Greg_S_1]

Hello.  I've been attempting to follow the instructions on this page https://software.intel.com/sites/products/documentation/hpc/ics/itac/81/Getting_Started.htm using Intel mpirun but have either not been able to get the trace files to be generated or I cannot find them.  An example of my command line input is:

mpirun -trace -n 8 <app> <parameters>

Is there something I'm missing, or are the trace files generated somewhere other than your working directory by default?

Thanks for any input.

[James_T_Intel]

Do you have Intel® Trace Analyzer and Collector installed?  Have you sourced the itacvars.sh script?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

[Greg_S_1]

Thanks for the quick response.  I have that software installed and I have sourced the file /opt/intel/itac/8.1.4.045/intel64/bin/itacvars.csh (using csh shell).

[James_T_Intel]

What is the output from the following:

env | grep I_MPI
env | grep VT
which mpirun

[Greg_S_1]

I_MPI_CC=icc
I_MPI_CXX=icc
I_MPI_F90=ifort

VT_MPI=impi4
VT_ROOT=/opt/intel//itac/8.1.4.045
VT_ARCH=intel64
VT_LIB_DIR=/opt/intel//itac/8.1.4.045/intel64/itac/lib_impi4
VT_SLIB_DIR=/opt/intel//itac/8.1.4.045/intel64/itac/slib_impi4
VT_MIC_SLIB_DIR=/opt/intel//itac/8.1.4.045/mic/itac/slib_impi4
VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread

/nisprod/products1/intelmpi/em64t/4.0.2.003/bin64/mpirun

[James_T_Intel]

Ok, I'll duplicate this and investigate this afternoon.  Is your application calling MPI_Init?  Also, try sourcing the /nisprod/products1/intelmpi/em64t/4.0.2.003/bin64/mpivars.sh script.

[Greg_S_1]

Yes, the application is calling MPI_Init.  I will try sourcing mpivars.sh and running again.  Are trace files generated as the application executes, or only after it finishes executing?  Thanks for your help so far.

[James_T_Intel]

The trace files are generated when MPI_Finalize is called.  If your application is failing, we have a failsafe option you can use as well, which will save data in a temporary file and write the trace file when the application ends in any manner.

[Greg_S_1]

Still no luck after sourcing mpivars.csh.  One thing to note is that our application does not call MPI_Finalize directly but calls Petsc_Finalize which in turn should call MPI_Finalize.

[James_T_Intel · Accepted solution]

I'm able to get the trace file using those versions and environment variables (or at least the equivalent for my different paths).  Try running this command:

LD_PRELOAD=libVT.so mpirun ...

Is there a <program>.prot file in the current folder?  This should be generated when MPI_Init is called if you are linking to the Trace Collector library.

[Greg_S_1]

Setting LD_PRELOAD to libVT.so seemed to do the trick.  Now both a set of .stf files and a .prot file are being generated (the .prot file wasn't being created before either).  Thanks for you help!