unable to LD_PRELOAD PMPI-based profiler with Fortran

[pcarns]

Hello,

I am trying to use an MPI profiling library with executables that have been built using Intel MPI. This particular library (Darshan 2.2.0-pre1) can be preloaded with LD_PRELOAD to intercept I/O related MPI function calls. It provides a wrapper for each function of interest to collect statistics and then invokes the PMPI version of each function so that the program operates as usual.

Everything works great with C programs or C++ programs, whether I use the Intel compilers or GNU compilers.

Unfortunately, I am having problems with Fortran. My main concern is with programs built with the Intel Fortran compiler, using either "mpiifort" or "mpif90 -fc=ifort". The executables work fine, but when I try to use the LD_PRELOAD'ed Darshan library it fails to intercept the underlying MPI calls. In fact, I can't even find any MPI functions in the symbols for the executable using gdb or nm, though obviously MPI is working fine in my test program.

Can someone help me figure out what I am doing wrong? Is there any way to intercept the MPI calls at run time from a Fortran program built using the Intel MPI suite?

To my knowledge this approach usuually works fine for Fortran programs built using MPI libraries based on MPICH or OpenMPI, although at least in the former case you normally have to preload an additional library (like libfmpich.so) for it to work properly. I tried preloading a few of the Intel .so libraries before the Darshan library in case there was a similar issue with the Intel suite, but I did not have any luck.

Many thanks!

[James_T_Intel]

Hi pcarns,

I'm currently looking into this issue. I have managed to replicate this on both the current release version of Darshan (2.1.2) and the pre-release version you are using. Are you able to intercept the calls if a different compiler is used with the Intel MPI Library?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[pcarns]

Thanks James.

Well, I didn't mention this in my original post in order to avoid comingingly what appeared to be two separate problems, but I actually cannot build my test program using Intel MPI in conjunction with the GNU fortran compilers. When I try to build the program using mpif90 I get a long list of undefined reference errors, starting with this (and covering a broad spectrum of MPI calls that aren't used directly in the test program):

[bash]/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_write_at_all' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_read_at_all_end' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Unpack_external' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Win_get_errhandler' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Type_contiguous' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Type_create_subarray' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Win_start'[/bash]

According to mpif90 --version, this is using "GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu3) 4.6.1".

For C and C++ example programs I can build fine with Inte MPI using either the Intel or GNU compilers with no problem (and the LD_PRELOAD works for either one).

Thanks,

-Phil

[pcarns]

Oh, and I should have mentioned that the Intel and GNU Fortran compilers are the only ones that I have available on the test workstation that I am using right now. I can't vouch for any of the other compilers.

Thanks,

-Phil

[James_T_Intel]

Hi Phil,

Which mpif90 are you using? The one provided with the Intel MPI Library should automatically link in all of the necessary libraries. It looks like you're not getting /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpi.so linked in, or not in the correct order. It should be linked before libmpigf.so is linked.

In my test, I am unable to intercept with gfortran as the compiler. I'll see if I can pin down exactly what's happening here.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[pcarns]

Hi James,

To be precise, here is what I see when I try to use mpif90 with the GNU fortran compiler:

[bash]pcarns@carns-x220:~/working/darshan-examples$ source ~/working/impi-4.0.3.008/bin64/mpivars.sh pcarns@carns-x220:~/working/darshan-examples$ which mpif90 /home/pcarns/working/impi-4.0.3.008/intel64/bin/mpif90 pcarns@carns-x220:~/working/darshan-examples$ mpif90 fperf.f -o fperf >& output.txt pcarns@carns-x220:~/working/darshan-examples$ head -n 4 output.txt /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_write_at_all' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_read_at_all_end' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Unpack_external' /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Win_get_errhandler' pcarns@carns-x220:~/working/darshan-examples$ mpif90 -show fperf.f -o fperf gfortran -ldl -ldl -ldl -ldl fperf.f -o fperf -I/home/pcarns/working/impi-4.0.3.008/intel64/include/gfortran -I/home/pcarns/working/impi-4.0.3.008/intel64/include -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /home/pcarns/working/impi-4.0.3.008/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.0.3 -lmpi -lmpigf -lmpigi -lpthread -lpthread -lpthread -lpthread -lrt[/bash]

The -ldl entries certainly look a little odd. I'll try tinkering with that underlying command line that is produced by the -show option and see if I can get it to work.

thanks,

-Phil

[pcarns]

I was able to get the GNU version to build by using the following manual command line rather than mpif90:

[bash]gfortran fperf.f -o fperf -I/home/pcarns/working/impi-4.0.3.008/intel64/include/gfortran -I/home/pcarns/working/impi-4.0.3.008/intel64/include -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -ldl -lmpigi -lmpigf -lmpi -lpthread -lrt[/bash] I think that the relevant part was to move the -lmpi argument so that it appears after the -lmpigf and -lmpigi arguments.

This executable also seems to run fine (just as when using the Intel Fortran compilers), but I still have the same problem as with the Intel compiler generated executables in that I can't figure out how to intercept the MPI functions.

thanks,

-Phil

[James_T_Intel]

Hi Phil,

Your compilation looks very similar to mine, with only a few differences in the include paths. For reference, here is what I get for the compilation command:

[bash][james@fxvm-jatullox01 io]$ mpif90 -show mpi_io.f90 -o test gfortran -ldl -ldl -ldl -ldl mpi_io.f90 -o test -I/opt/intel/impi/4.0.3.008/intel64/include/gfortran/4.4.0 -I/opt/intel/impi/4.0.3.008/intel64/include -L/opt/intel/impi/4.0.3.008/intel64/lib -L/opt/intel/impi/4.0.3.008/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/impi/4.0.3.008/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.0.3 -lmpi -lmpigf -lmpigi -lpthread -lpthread -lpthread -lpthread -lrt[/bash]
I am able to compile with no errors or warnings.

As far as intercepting the calls, I am contacting our developers for additional information.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[pcarns]

Hi James,

Thank you very much, and I appreciate you trying it out first hand.

thanks,

-Phil

[James_T_Intel]

Hi Phil,

Just out of curiosity, when you are attempting to intercept the Fortran calls, are you accounting for the name differences somewhere? Looking through the Darshan code I didn't see anywhere that this was done. As an example, look at MPI_FILE_WRITE_ORDERED. In C/C++, the name of the function is MPI_File_write_ordered. In Fortran (if seen from C), it is mpi_file_write_ordered_. That could be what you are encountering. Here is a small C++ code I used to intercept a Fortran call to MPI_FILE_WRITE_ORDERED.

[cpp]

#include #include extern "C" {void mpi_file_write_ordered_(MPI_Fint, void*, MPI_Fint, MPI_Fint, MPI_Fint*, MPI_Fint);} void mpi_file_write_ordered_(MPI_Fint fh_, void* buf, MPI_Fint count, MPI_Fint datatype_, MPI_Fint *status_, MPI_Fint ierror) { std::cout << "Got it in C++" << std::endl; MPI_File fh=MPI_File_f2c(fh_); MPI_Datatype datatype=MPI_Type_f2c(datatype_); MPI_Status *status=new MPI_Status; ierror=MPI_File_write_ordered(fh,but,count,datatype,status); fh_=MPI_File_c2f(fh); if (status_ != MPI_F_STATUS_IGNORE && status_ != MPI_F_STATUSES_IGNORE) { MPI_Status_c2f(status,status_); } return; }[/cpp]

I was able to preload the library generated by this and intercept a Fortran call to MPI_FILE_WRITE_ORDERED with no problem.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[pcarns]

Thanks James. I won't have time to look at this for a few days, but I will try it out later this week and report back.

thanks,

-Phil

[pcarns]

Hi James,

I haven't tried LD_PRELOAD wrapper yet, but I did confirm that I can set debugger breakpoints on functions following that naming convention (like mpi_file_open_). I have a few follow up questions, though, because this wasn't the profiling interface that I expected:

• Is there any shared entry point for C and Fortran bindings, or should a wrapper for the former catch MPI_File_open() while a wrapper for the latter catches mpi_file_open_()?
• Do these mpi_file_open_ etc. function names have exactly the same arguments as the C interface functions like MPI_File_open()?
• If I make a wrapper for mpi_file_open_, can it invoke an underlying pmpi_file_open_ after collecting profiling information? In other words, does it follow the convention of the MPI/PMPI function names in the C interface?
• Is the fortran function naming convention fairly stable across MPI releases (i.e., if one were to generate wrappers using those function names, will they likely still work in future Intel MPI releases?)?

Thanks! -Phil

[James_T_Intel]

Hi Phil,

Shared entry points: I will need to check with the developers on this point, but for a more general use, I would recommend a separate interceptor for each.

Function arguments: No. In almost all MPI functions the Fortran call has an additional argument at the end for the return code. Also, the Fortran MPI interface is implemented mostly as subroutines with no return value rather than functions.

MPI/PMPI: I'll need to verify this with the developers as well. However, I believe that it will follow the same naming convention. I just modified the code I posted earlier to call PMPI_File_write_ordered instead. However, there are additional considerations I will mention shortly.

Naming convention: It is (should be) as stable as the MPI Standard, at least with regards to the MPI call structure.

Now, here's something to keep in mind. I don't know how much Fortran you use, so I'm going to start at a pretty basic level, forgive me if I sound like I'm lecturing. Fortran is case insensitive, and the names used in Fortran are mangled by the compiler to account for this. However, nothing defineshow a compiler should mangle names, and frequently compilers use different name mangling methods. These can also be configured via command line options at compile time. As a quick example, MPI_File_open in Fortran could really end up in any of these forms:

[plain]MPI_FILE_OPEN mpi_file_open MPI_FILE_OPEN_ mpi_file_open_ MPI_FILE_OPEN__ mpi_file_open__[/plain]
Possibly others depending on the exact compiler and options (these are the naming variants with which I'm most familiar). If you stay within one language, this distinction is mostly irrelevant. However, you are mixing languages, and thus it must be considered.

Hopefully that's clear. I'll check with the developers on some of the specifics, and let you know when I've got some more information.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[pcarns]

Ah, thanks for the clarification James. That makes sense. It didn't occur to me that I was just looking at the mangled version of the Fortran functions. I am definitely a novice at Fortran development!

These Fortran functions/symbols are not what I was hoping for, though. In other major MPI implementations, there is at some level a set of C functions (usually it is just the normal mpi.h-defined C functions for the routines in question) that all bindings use under the covers. In other words, the underlying library is implemented as a collection of native C functions, and the various bindings are just invoking wrappers that account for argument differences and name mangling before calling those C functions. The side effect of this organization is that if you provide profiling wrappers for that C interface, you automatically catch the C++ and Fortran programs as well because they are funneling through that same underlying API. This may not be the case in Intel MPI, though.

thanks,

-Phil

[pcarns]

I found some more information. I _can_ set a breakpoint on the PMPI_* version of the C functions rather than the MPI_* version of the C functions. For example:

[bash]Breakpoint 3, 0x00007ffff7916330 in PMPI_File_open () from /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpi.so.4 (gdb) where #0 0x00007ffff7916330 in PMPI_File_open () from /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpi.so.4 #1 0x00007ffff7ec79b4 in pmpi_file_open__ () from /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so.4 #2 0x000000000040383c in MAIN__ ()[/bash]

Does this mean that the Fortran bindings are calling the PMPI functions directly instead of the intermediate MPI functions? I'm not sure why that would be the case, but it would explain the behavior that I am seeing. thanks, -Phil

[James_T_Intel]

Hi Phil,

You will need to write a separate interface function for the C and Fortran calls. The Fortran and C functions will both call the same PMPI version for the actual work. This is similar to what the Intel Trace Analyzer and Collector does for profiling.

In order to simplify what you are doing, you could have your Fortran intercept call the C MPI API function (after appropriate conversions) and do the profiling completely within the C version. That will allow you to only maintain one set of profiling functionality, with two interfaces.

I hope this helps clarify what you'll need to do. Do you have any otherquestions?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools