problem on fortran + open MPI on a first principle calculation program

[Hiroshi_Suzuki]

I am trying to use Phase program (first principle calculation package) on Ubuntu 8.04 with Intel compiler 11.1 (build 072) and mkl 10.2.5 (build 035)

Serial (non-mpi) version of the program was successfully compiled with the following options in a Makefile:

F90FLAGS = -w90 -w95 -W0 -traceback -I./no_mpi/include
F77FLAGS = -w90 -w95 -W0 -traceback -I./no_mpi/include
CFLAGS = -O -DINTEL
CPPFLAGS = -DLinux -DWF_JRCATFFT_WS -DCD_JRCATFFT_WS -D_NO_MPI_ -D_NO_ARG_DUMMY_ -D_POT_SMOOTHING_ -DTRANSPOSE -DGGA_ATOMIC_WITH_NEW_GNCPP -DREMOVE_PC_FROM_FORCE -D_HEAP_SORT_ -D_FAST_WAY_
LFLAGS = -i-static -Bstatic
LIBS = -L/opt/intel/mkl/10.2.5.035/lib/32 -lmkl_lapack95 -lmkl_solver_sequential -Wl,--start-group /opt/intel/mkl/10.2.5.035/lib/32/libmkl_intel.a /opt/intel/mkl/10.2.5.035/lib/32/libmkl_sequential.a /opt/intel/mkl/10.2.5.035/lib/32/libmkl_core.a -Wl,--end-group -lguide -lpthread

The program works fine.

Next I installed open MPI 1.4.2 to /opt/ompi. The package installed successfully, and sample programs worked correctly.

Then I proceed to use mpi with Phase program, try to compile with the following options:

F90FLAGS = -w90 -w95 -W0 -traceback -I/opt/ompi/include
F77FLAGS = -w90 -w95 -W0 -traceback -I/opt/ompi/include
CFLAGS = -O -DINTEL
CPPFLAGS = -DLinux -DFFTW3 -D_POT_SMOOTHING_ -DTRANSPOSE -DGGA_ATOMIC_WITH_NEW_GNCPP -DREMOVE_PC_FROM_FORCE -D_HEAP_SORT_ -D_FAST_WAY_
LFLAGS = -i-static -Bstatic
LIBS = -L/opt/intel/mkl/10.2.5.035/lib/32 -lmkl_lapack95 -lmkl_solver -Wl,--start-group /opt/intel/mkl/10.2.5.035/lib/32/libmkl_intel.a /opt/intel/mkl/10.2.5.035/lib/32/libmkl_intel_thread.a /opt/intel/mkl/10.2.5.035/lib/32/libmkl_core.a /opt/intel/mkl/10.2.5.035/lib/32/libmkl_blacs_openmpi.a -Wl,--end-group -lfftw3xf_intel -lguide -lpthread -Bdynamic -L/opt/ompi/lib -lmpi

Compiling stops with the following messages (only first several lines are shown) trying to build an executable.

m_Parallelization.o: In function `m_parallelization_mp_m_parallel_get_nproc_from_arg_':
m_Parallelization.F90:(.text+0x7a5): undefined reference to `mpi_bcast_'
m_Parallelization.F90:(.text+0x7d8): undefined reference to `mpi_bcast_'
m_Parallelization.o: In function `m_parallelization_mp_m_parallel_end_mpi_':
m_Parallelization.F90:(.text+0x1009): undefined reference to `mpi_finalize_'
m_Parallelization.o: In function `m_parallelization_mp_m_parallel_init_mpi_gga_':
m_Parallelization.F90:(.text+0x7d3a): undefined reference to `mpi_comm_split_'
m_Parallelization.F90:(.text+0x7d59): undefined reference to `mpi_comm_size_'

It looks like the linker could not find mpi library. What is wrong with my Makefile and how to link open MPI with my phase program?

[TimP]

Did you rebuild openmpi from source code to use ifort?
e.g.
export FC=ifort
export F77=ifort
../configure --prefix=/opt/ompi
make
make install

This would set up MPI to use gcc/g++/ifort. Don't use /opt/ompi for this build if the gnu version of OpenMPI is already there; I would suggest something like /opt/ompiifort.
Or, did you install a pre-built openmpi set up for the gnu compilers supplied with Ubuntu?
The mpif77 and mpif90 wrappers set up the ifort command to include the MPI include and library paths. Your Makefile should be using the wrappers to link, not the bare compiler or ld.
I've never heard of anyone trying MPI on 32-bit Ubuntu, but there's no reason it shouldn't be possible.
Thanks to Ron's reference at the top of this forum, ifort works well with 64-bit Ubuntu.

[Hiroshi_Suzuki]

Thank you for a followup.

Yes, I build open MPI from source tarball as the following:

./configure --prefix=/opt/ompi CC=icc CXX=icpc F77=ifort FC=ifort
make all
make install

Then set environment in .bashrc as:
MPIROOT=/opt/ompi
PATH=$MPIROOT/bin:$PATH
LD_LIBRARY_PATH=$MPIROOT/lib:$LD_LIBRARY_PATH
MANPATH $MPIROOT/man:$MANPATH
export MPIROOT PATH LD_LIBRARY_PATH MANPATH

Create openmpi.conf in /etc/ld.so.conf.d which consists of:
/opt/ompi/lib

Refresh shared library with:
ldconfig

Create ~/.rhosts which consists of:
hostname (my PC's hostname)

Create ~/.hosts which consists of:
hostname cpu=2 (my PC has Core2Duo CPU)

Above is in accordance with instructions in open MPI installation manual.

Build example programs in examples directory with:
make all

Every program works fine.

[Hiroshi_Suzuki]

Problem solved.

First, I switch to 64 bit version of Ubuntu 9.04 because the system I am trying is more suitable for 64 bit OS. Then I take the following steps:

1) give up to use MKL and also stop using FFT supplied with MKL. Instead, I use BLAS/LAPACK and FFT library comes with the software I am going to use. These makes LIBS line in Makefile quite simple as the follows:

LIBS = -L./ -llapack -lblas -Bdynamic -L/opt/ompi/lib -lmpi

where /opt/ompi is the directory I install open MPI.

2) change to mpif90 and mpicc wrapper command for F90 and CC.

F90 = mpif90
CC = mpicc

3) also change LINK line to mpif90

LINK = mpif90

4) add "-shared-intel" option to F90FLAGS, F77FLAGS and CFLAGS.

5) finally add the following line to /etc/ld.so.conf. I already add openmpi.conf file that directs library of open MPI to /etc/ld.so.conf.d, therefore this may not necessary, in case the position of open MPI library does not pass to the software correctly.

include /opt/ompi/lib

The Phase first principle calculation package finally start working on my 8 cpus!