Error whem compiling MPI using mpiifort

[Oswaldo_D_]

Hello,

I am trying to compile a program to run under an multi-core system. I have set all the dependencies and ran all the setup scripts for the environment variables in order to compile the program using the intel fortran compiler and the intel mpiifort. The main program and some of the subroutines compile fine (giving the 13003 warning) but when the dependencies are going to be resolved for the MPI interfaces I get the following messages:

/apps/3rdparty/intel/icsxe/composer_xe_2013_sp1.2.144/bin/intel64/ifort -O3 -module /glb/home/usocab/onshore/AASPI/include64 -check bounds -traceback -heap-arrays -I/glb/home/usocab/onshore/AASPI/include64 -I/glb/home/usocab/onshore/AASPI/ext_lib64/intel_shell/includes -c -o obj64/get_lags_ow.o get_lags_ow.f90
warning #13003: message verification failed for: 28003; reverting to internal message
warning #13003: message verification failed for: 28008; reverting to internal message
/apps/3rdparty/intel/icsxe/composer_xe_2013_sp1.2.144/bin/intel64/ifort -O3 -module /glb/home/usocab/onshore/AASPI/include64 -check bounds -traceback -heap-arrays obj64/dip3d.o obj64/get_pq_inv.o obj64/get_lags_ow.o -L/glb/home/usocab/onshore/AASPI/lib64 -laaspi_util_new_reorg -laaspi_io -lstdc++ -I/glb/home/usocab/onshore/AASPI/ext_lib64/intel_shell/lib -mkl -mkl -o obj64/dip3d
/glb/apps/3rdparty/intel/icsxe/composer_xe_2013_sp1.2.144/compiler/lib/intel64/for_main.o: In function `main':
/export/users/nbtester/efi2linux_nightly/branch-14_0/20140121_010000/libdev/frtl/src/libfor/for_main.c:(.text+0x31): undefined reference to `__intel_new_feature_proc_init'
obj64/dip3d.o: In function `MAIN__':
dip3d.f90:(.text+0x3e): undefined reference to `__intel_new_feature_proc_init'
dip3d.f90:(.text+0x461): undefined reference to `mpi_bcast_'
dip3d.f90:(.text+0xfdf): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0x103e): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0xbfa0): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0x14917): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0x160e2): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0x16162): undefined reference to `mpi_finalize_'
dip3d.f90:(.text+0x2eebf): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0x2eee6): undefined reference to `mpi_bcast_'
dip3d.f90:(.text+0x2ef0d): undefined reference to `mpi_bcast_'
dip3d.f90:(.text+0x2ef34): undefined reference to `mpi_bcast_'
dip3d.f90:(.text+0x2ef5b): undefined reference to `mpi_bcast_'
dip3d.f90:(.text+0x2ef82): undefined reference to `mpi_bcast_'
obj64/dip3d.o:dip3d.f90:(.text+0x2efa9): more undefined references to `mpi_bcast_' follow
obj64/dip3d.o: In function `MAIN__':
dip3d.f90:(.text+0x31c91): undefined reference to `mpi_barrier_'
dip3d.f90:(.text+0x31cb8): undefined reference to `mpi_bcast_'
dip3d.f90:(.text+0x31cdf): undefined reference to `mpi_bcast_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(mpi_send_receive_array_util.o): In function `send_real_array':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/mpi_send_receive_array_util.f90:64: undefined reference to `mpi_send_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(mpi_send_receive_array_util.o): In function `send_logical_array':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/mpi_send_receive_array_util.f90:116: undefined reference to `mpi_send_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(mpi_send_receive_array_util.o): In function `receive_real_array':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/mpi_send_receive_array_util.f90:168: undefined reference to `mpi_recv_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(mpi_send_receive_array_util.o): In function `receive_logical_array':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/mpi_send_receive_array_util.f90:221: undefined reference to `mpi_recv_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(mpi_send_receive_array_util.o): In function `receive_integer_array':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/mpi_send_receive_array_util.f90:275: undefined reference to `mpi_recv_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(mpi_send_receive_array_util.o): In function `send_integer_array':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/mpi_send_receive_array_util.f90:330: undefined reference to `mpi_send_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(prog_util.o): In function `check_for_errors_found_only_on_master':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:82: undefined reference to `mpi_bcast_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:92: undefined reference to `mpi_barrier_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:93: undefined reference to `mpi_finalize_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(prog_util.o): In function `check_for_errors_found_on_any_processor':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:162: undefined reference to `mpi_recv_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:175: undefined reference to `mpi_send_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:184: undefined reference to `mpi_bcast_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:196: undefined reference to `mpi_abort_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(prog_util.o): In function `sum_values_from_each_processor':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:258: undefined reference to `mpi_recv_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:268: undefined reference to `mpi_send_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(prog_util.o): In function `check_errors':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:322: undefined reference to `mpi_bcast_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:330: undefined reference to `mpi_barrier_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:331: undefined reference to `mpi_finalize_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(prog_util.o): In function `check_for_processor_errors':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:399: undefined reference to `mpi_recv_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:412: undefined reference to `mpi_send_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:421: undefined reference to `mpi_bcast_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:431: undefined reference to `mpi_abort_'
/glb/home/usocab/onshore/AASPI/lib64/libaaspi_util_new_reorg.a(prog_util.o): In function `invoke_mpi':
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:553: undefined reference to `mpi_init_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:558: undefined reference to `mpi_comm_rank_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:563: undefined reference to `mpi_comm_size_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:576: undefined reference to `mpi_bcast_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:577: undefined reference to `mpi_barrier_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:585: undefined reference to `mpi_barrier_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:586: undefined reference to `mpi_get_processor_name_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:588: undefined reference to `mpi_send_'
/glb/am/sepco/proj/us_onshore_reg/AASPI/src/lib_new_reorg/prog_util.f90:594: undefined reference to `mpi_recv_'
make: *** [obj64/dip3d] Error 1

Any help is very much appreciated.

Best regards,

Oswaldo

[Kevin_D_Intel]

I don't know whether this helps any. It is not very clear what might be causing all that you are seeing.

The warnings regarding reverting to the internal message may suggest an environment setup issue. I know you said you ran the setup scripts but those messages can occur where the compilervars script has not been run.

Is there a reason why you used the full-path to the compiler or is that just a by-product of some screen capture output?

In the command-line performing the link, you have what appears to be a library path, /glb/home/usocab/onshore/AASPI/ext_lib64/intel_shell/lib, specified with either a lower-case L or an upper case I, I can't tell for sure. For a library path it should be specified using the capital L (e.g. -L<path>).

Was the link actually issued by the mpiifort script or had you run 'ifort' directly as shown in what you posted?
 
The warnings and unresolved externals feel like an environment setup issue.

[Oswaldo_D_]

Thanks so much Kevin for your input. Let me send you the Makefile.inc so you can see what's going on there. When I compile the code everything is sourced from that Makefile.inc. The path is specfified on the makefile with an capital i (I).

I wrote a message to Ron Green about it and he suggested that I should change the compiler from ifort to mpiifort. I did so and all the error messages went away but now the code does not work when I submit it as a job on our LSF system. I have attached the output error file so you can take a look at it also.

Any help is very much appreciated!

Best regards,

Oswaldo

[Kevin_D_Intel]

If the app you are building really consists of Fortran, C, and C++, then it would appear it may be using a mix of compilers based on these from your Makefile.inc file:

CC_64= /apps/3rdparty/intel/ics2013/composer_xe_2013.0.060/bin/intel64/icc
CXX_64= g++ -m64
F90_64= /apps/3rdparty/intel/impi/4.1.0.017/intel64/bin/mpiifort

Given what you have here leads me to believe you perhaps have not sourced the proper compiler setup scripts but maybe invoking them directly and that may be the cause of the MPI failure.

Unless there’s some reason for using g++, what I recommend you try using are these definitions:

CC_64= mpiicc
CXX_64= mpiicpc
F90_64= mpiifort

Then before running the Makefile, source the compilervars.sh found under the compiler installation “bin” directory and then source the Intel MPI mpivars.sh setup and then ensure the compilers are accessible via these commands:

ifort -V
icc -V
icpc -V

I would do a complete clean of your application (e.g. make clean or whatever the app provides), then try the make again.

From information that I’m finding about similar errors to the run-time errors you provided, others suggested those can appear when mixing non-compatible MPI implementations. For example, using the incorrect mpi.h for the MPI software used. Maybe somehow you are mixing up MPI with the mix of compilers that you setup in your Makefile.inc.

There is an Intel MPI Getting Started Guide also here: https://software.intel.com/en-us/node/498676

[Oswaldo_D_]

Thank so much Kevin, this really helped moving forward. I made sure to run the compilervars.sh and the Intel MPI mpivars.sh before running the make file. I also, followed your suggestion of a make clean. I have changed the the compiler definitions per you suggestion. when I type your -V suggestions above I get:

/home/usocab% ifort -V
Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.144 Build 20140120
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

/home/usocab% icc -V
Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.144 Build 20140120
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

/home/usocab% icpc -V
Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.144 Build 20140120
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

So, I guess everything is fine on that side. I got all the programs to compile ok, now when I submit to the LSF system I get an old error, the PMPI_Comm_rank: Invalid communicator but before that I get this new error:

librdmacm: Warning: couldn't read ABI version.
librdmacm: Warning: assuming: 4
librdmacm: Fatal: unable to get RDMA device list

See the attached output file. Once again thanks so much for your help!

Best regards,

Oswaldo