I am trying to use an MPI profiling library with executables that have been built using Intel MPI. This particular library (Darshan 2.2.0-pre1) can be preloaded with LD_PRELOAD to intercept I/O related MPI function calls. It provides a wrapper for each function of interest to collect statistics and then invokes the PMPI version of each function so that the program operates as usual.
Everything works great with C programs or C++ programs, whether I use the Intel compilers or GNU compilers.
Unfortunately, I am having problems with Fortran. My main concern is with programs built with the Intel Fortran compiler, using either "mpiifort" or "mpif90 -fc=ifort". The executables work fine, but when I try to use the LD_PRELOAD'ed Darshan library it fails to intercept the underlying MPI calls. In fact, I can't even find any MPI functions in the symbols for the executable using gdb or nm, though obviously MPI is working fine in my test program.
Can someone help me figure out what I am doing wrong? Is there any way to intercept the MPI calls at run time from a Fortran program built using the Intel MPI suite?
To my knowledge this approach usuually works fine for Fortran programs built using MPI libraries based on MPICH or OpenMPI, although at least in the former case you normally have to preload an additional library (like libfmpich.so) for it to work properly. I tried preloading a few of the Intel .so libraries before the Darshan library in case there was a similar issue with the Intel suite, but I did not have any luck.Many thanks!
Just out of curiosity, when you are attempting to intercept the Fortran calls, are you accounting for the name differences somewhere? Looking through the Darshan code I didn't see anywhere that this was done. As an example, look at MPI_FILE_WRITE_ORDERED. In C/C++, the name of the function is MPI_File_write_ordered. In Fortran (if seen from C), it is mpi_file_write_ordered_. That could be what you are encountering. Here is a small C++ code I used to intercept a Fortran call to MPI_FILE_WRITE_ORDERED.[cpp]
I was able to preload the library generated by this and intercept a Fortran call to MPI_FILE_WRITE_ORDERED with no problem.
Shared entry points: I will need to check with the developers on this point, but for a more general use, I would recommend a separate interceptor for each.
Function arguments: No. In almost all MPI functions the Fortran call has an additional argument at the end for the return code. Also, the Fortran MPI interface is implemented mostly as subroutines with no return value rather than functions.
MPI/PMPI: I'll need to verify this with the developers as well. However, I believe that it will follow the same naming convention. I just modified the code I posted earlier to call PMPI_File_write_ordered instead. However, there are additional considerations I will mention shortly.
Naming convention: It is (should be) as stable as the MPI Standard, at least with regards to the MPI call structure.
Now, here's something to keep in mind. I don't know how much Fortran you use, so I'm going to start at a pretty basic level, forgive me if I sound like I'm lecturing. Fortran is case insensitive, and the names used in Fortran are mangled by the compiler to account for this. However, nothing defineshow a compiler should mangle names, and frequently compilers use different name mangling methods. These can also be configured via command line options at compile time. As a quick example, MPI_File_open in Fortran could really end up in any of these forms: