compiling problem: libmpi_mt.so vs libmpi.so

[Beichuan_Y_]

Hi,

1. My hybrid mpi/openmpi code compiled with intelmpi-4.1.3.048 and intel-compilers-13.0.1 gives the following errors:

mpiicpc -o paraEllip3d -O3 -DNDEBUG -openmp -mt_mpi  Assembly.o Boundary.o Contact.o Cylinder.o Fluid.o Parameter.o Particle.o Rectangle.o Spring.o Vec.o balanc.o const.o hqr.o main.o nrutil.o ran.o realtypes.o root6.o zrhqr.o -I/work4/projects/openmpi/boost-1.53.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/include -L/work4/projects/openmpi/boost-1.53.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/lib -lboost_mpi -lboost_serialization -lboost_timer -Wl,-rpath=/work4/projects/openmpi/boost-1.53.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/lib

ld: MPIR_Thread: TLS definition in /apps/intel/impi/4.1.3.048/intel64/lib/libmpi_mt.so section .tbss mismatches non-TLS definition in /apps/intel/impi/4.1.3.048/intel64/lib/libmpi.so.4 section .bss
/apps/intel/impi/4.1.3.048/intel64/lib/libmpi.so.4: could not read symbols: Bad value
make: *** [paraEllip3d] Error 1

2. However, it compiles and runs well with intelmpi/4.0.3.008 and intel-compilers/12.1.0. So I am wondering if there is anything wrong with intelmpi-4.1.3.048 and intel-compilers-13.0.1.

Note that "-openmp -mt_mpi" option is used for both cases.

3. FYI, my code uses C++ boost.mpi libraries, which are compiled well with both intelmpi/4.0.3.008 and intel-compilers/12.1.0, and intelmpi-4.1.3.048 and intel-compilers-13.0.1, so I don't think boost.mpi causes this problem.

Thanks,

Beichuan Yan

[Beichuan_Y_]

mpiicpc -show:

icpc -I/apps/intel/impi/4.1.3.048/intel64/include -L/apps/intel/impi/4.1.3.048/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /apps/intel/impi/4.1.3.048/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.1 -lmpigc4 -lmpigf -lmpi -lmpigi -ldl -lrt -lpthread

[James_T_Intel]

Hi Beichuan,

When you compiled the Boost MPI libraries, did you compile with -mt_mpi?  Somewhere in your compilation, you are pulling in the single-threaded MPI library, which is conflicting with the multi-threaded library.  Try adding -show to the full link command (and the compile steps) and see if that shows where the incompatible library is pulled in.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

[Beichuan_Y_]

James,

Thanks. Look at the mpiicpc -show:

icpc -I/apps/intel/impi/4.1.3.048/intel64/include -L/apps/intel/impi/4.1.3.048/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /apps/intel/impi/4.1.3.048/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.1 -lmpigc4 -lmpigf -lmpi -lmpigi -ldl -lrt -lpthread

How to make this mpiicpc use -lmpi_mt instead of -lmpi? should the sysadmin rebuild intel mpi with a smart choice between libmpi and libmpi_mt?

FYI, this issue causes problem on HDF5: in http://www.prace-project.eu/IMG/pdf/d7.2.2.pdf , which says:

The parallel HDF5 library is usually linked with the MPI-compiler wrappers which, for Intel MPI leads to a dependency on the plain MPI runtime library (libmpi), or on its multithreaded version (libmpi_mt). Starting with version 4.1.0 of the Intel MPI library, this causes a conflict when trying to link a multithreaded code with the plain-MPI variant of the HDF5 library, or vice versa. Together with Intel, RZG has worked out a patch for the HDF5 installation procedure which simply accounts for using the plain linker (instead of the MPI wrapper) for building HDF5. This makes a single HDF5 installation compatible with both, plain MPI and hybrid (MPI + threads) codes.

Thanks,

Beichuan

[James_T_Intel]

If you just use mpiicpc -show, there is nothing indicating that mpi_mt should be linked.  Instead, add -show to the full link command, where you have -mpi_mt.  This will add libmpi_mt to the link command.

[Beichuan_Y_]

Thanks. I am a little confused by how to add -show to full link command, can you please look at my makefile, and should I do  MPICXX = mpiicpc -show instead?

#########################################################
###        Part A: change with your environment       ###
#########################################################
###1. system platform
PLATFORM = supercomputer
#PLATFORM = workstation

###2. compilers package
#COMPILERS = gcc
COMPILERS = intel

###3. boost location
ifeq ($(PLATFORM), supercomputer)
 ifeq ($(COMPILERS), gcc)
  BOOST_ROOT = /work4/projects/openmpi/boost-1.53.0-openmpi-1.7.5-gcc-compilers-4.7.3
 endif
 ifeq ($(COMPILERS), intel)
  #BOOST_ROOT = /work4/projects/openmpi/boost-1.53.0-intelmpi-4.0.3.008
  BOOST_ROOT = /work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1
 endif
endif

ifeq ($(PLATFORM), workstation)
 BOOST_ROOT = /usr/local/boost-1.53.0-openmpi-1.6.4-gcc-4.6.4
 #BOOST_ROOT = default
endif

#########################################################
###         Part B: usually do not need change        ###
#########################################################
###1. MPICXX, OPENMPI and OPTIMIZE
# C++ compiler
ifeq ($(COMPILERS), gcc)
 MPICXX = mpic++
endif

ifeq ($(COMPILERS), intel)
 MPICXX = mpiicpc
endif

# OpenMP
ifeq ($(COMPILERS), gcc)
 OPENMP = -fopenmp
endif

ifeq ($(COMPILERS), intel)
 OPENMP = -openmp -mt_mpi
endif

# optimize or debug
ifeq ($(COMPILERS), gcc)
 #full level of IEEE 754 compliance
 #OPTIMIZE = -O3 -DNDEBUG -frounding-math -fsignaling-nans
 OPTIMIZE = -O3 -DNDEBUG
endif
ifeq ($(COMPILERS), intel)
 #-fp-model source also implies keyword precise; and it is nearly as twice slow as default -fp-model fast=1
 #OPTIMIZE = -O3 -DNDEBUG -fp-model source
 OPTIMIZE = -O3 -DNDEBUG
endif
#OPTIMIZE = -Wall -g

# GCC quadmath, default off
#QUADMATH = -DQUADMATH -L/usr/local/gcc-4.6.2/lib64 -lquadmath

# CXXFLAGS
CXXFLAGS = $(OPTIMIZE) $(OPENMP) $(QUADMATH)

###2. BOOST details
ifeq ($(PLATFORM), supercomputer)
 BOOST_INCLUDE=-I$(BOOST_ROOT)/include
 BOOST_LIB_DIR=-L$(BOOST_ROOT)/lib
 BOOST_LIBS=-lboost_mpi -lboost_serialization -lboost_timer
 BOOST_RUN_LIB_DIR=-Wl,-rpath=$(BOOST_ROOT)/lib
endif

ifeq ($(PLATFORM), workstation)
  ifeq ($(BOOST_ROOT), default)
   BOOST_LIB_DIR=-L/usr/lib64/openmpi/lib:/usr/lib64/compat-openmpi/lib:/usr/lib64
   BOOST_LIBS=-lboost_mpi -lboost_serialization -lboost_timer
   BOOST_RUN_LIB_DIR=-Wl,-rpath=/usr/lib64/openmpi/lib:/usr/lib64/compat-openmpi/lib:/usr/lib64
  else
   BOOST_INCLUDE=-I$(BOOST_ROOT)/include
   BOOST_LIB_DIR=-L$(BOOST_ROOT)/lib
   BOOST_LIBS=-lboost_mpi -lboost_serialization -lboost_timer
   BOOST_RUN_LIB_DIR=-Wl,-rpath=$(BOOST_ROOT)/lib
  endif
endif

INCLUDE=$(BOOST_INCLUDE)
LIB_DIR=$(BOOST_LIB_DIR)
LIBS=$(BOOST_LIBS)
RUN_LIB_DIR=$(BOOST_RUN_LIB_DIR)

###3. makefile
SOURCES = $(wildcard *.cpp)
OBJECTS = $(SOURCES:.cpp=.o)
#OBJECTS = $(patsubst %.cpp, %.o, $(SOURCES))
EXECUTABLE = paraEllip3d

.PHONY: all tar clean

all: $(EXECUTABLE)

$(EXECUTABLE): $(OBJECTS)
        $(MPICXX) -o $@ $(CXXFLAGS) $(OBJECTS) $(INCLUDE) $(LIB_DIR) $(LIBS) $(RUN_LIB_DIR)

%.o: %.cpp
        $(MPICXX) -c $< -o $@ $(CXXFLAGS) $(INCLUDE) $(LIB_DIR) $(LIBS)

sinclude $(SOURCES:.cpp=.d)

%.d: %.cpp
        $(MPICXX) -MM $< > $@.$$$$; \
        sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@; \
        rm -f $@.$$$$

tar:
        tar -cvf $(EXECUTABLE).tar *.h *.cpp makefile* readme

clean:
        -rm -f *.o *.d  *~ *.tar $(EXECUTABLE)

# sinclude is always resolved even if make tar/clean

[James_T_Intel]

The easiest would be to add it to CXXFLAGS.  Also, when you compiled the Boost* libraries, did you use -mt_mpi here?

[Beichuan_Y_]

Thanks. Boost could be a little tricky, I will rebuild it with:

using mpi : /app/intel/impi/4.1.3.048/bin64/mpiicpc : <cxxflags>-lmpi_mt <linkflags>-lmpi_mt ;

If you have experience on building boost with -mt_mpi please advise, thanks.

[Beichuan_Y_]

1. I rebuild boost-1.55.0 with the command: ./b2 toolset=intel cflags="-lmpi_mt" cxxflags="-lmpi_mt" linkflags="-lmpi_mt" install

2. Then I compiled my code by adding "-show" to my CXXFLAGS, it just shows the following and does not compile:

mpiicpc -o paraEllip3d -show -O3 -DNDEBUG -openmp -mt_mpi  Assembly.o Boundary.o Contact.o Cylinder.o Fluid.o Parameter.o Particle.o Rectangle.o Spring.o Vec.o balanc.o const.o hqr.o main.o nrutil.o ran.o realtypes.o root6.o zrhqr.o -I/work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/include -L/work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/lib -lboost_mpi -lboost_serialization -lboost_timer -Wl,-rpath=/work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/lib

icpc -o paraEllip3d -O3 -DNDEBUG -openmp Assembly.o Boundary.o Contact.o Cylinder.o Fluid.o Parameter.o Particle.o Rectangle.o Spring.o Vec.o balanc.o const.o hqr.o main.o nrutil.o ran.o realtypes.o root6.o zrhqr.o -I/work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/include -L/work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/lib -lboost_mpi -lboost_serialization -lboost_timer -Wl,-rpath=/work4/projects/openmpi/boost-1.55.0-intelmpi-4.1.3.048-intel-compilers-13.0.1/lib -I/apps/intel/impi/4.1.3.048/intel64/include -L/apps/intel/impi/4.1.3.048/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /apps/intel/impi/4.1.3.048/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.1 -lmpigc4 -lmpigf -lmpi_mt -lmpigi -ldl -lrt -lpthread

3. Then I removed "-show" and now it compiles.

Thanks,

Beichuan