Compilation errors: libmpi_mt.so and libmpi.so.4 mismatch

[ffgarcia]

Hi,

the MPI compilation of my code results in the following error:

ld: MPIR_Thread: TLS definition in /opt/apps/intel13/impi/4.1.0.030/intel64/lib/libmpi_mt.so section .tbss mismatches non-TLS definition in /opt/apps/intel13/impi/4.1.0.030/intel64/lib/libmpi.so.4 section .bss

/opt/apps/intel13/impi/4.1.0.030/intel64/lib/libmpi.so.4: could not read symbols: Bad value

I have sent the systems admin. several e-mails but he just can't seem to figure it out. Any help on how to fix this problem would be appreciated.

Thank you.

[James_T_Intel]

Hi,

How are you compiling and linking?  At a guess, it looks like you are mixing the standard and thread-safe MPI libraries.  If you are using the compiler scripts, look for -mt_mpi, -openmp, -parallel, -threads, -reentrancy, or -reentrancy threaded.  Any of these options will link to the thread-safe library (libmpi_mt).  If you are using them, make certain that they are used on all compliation and linking commands.  If you are using one of the flags other than -mt_mpi and want to restrict it to only the necessary routines, then adding -mt_mpi to others should be sufficient.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

[tanksnr]

Hi,

Hope you all don't mind me resurrecting this one.

But I have the exact same, or at least similar, error.

ld: MPIR_Thread: TLS definition in /opt/gridware/c8000/intel/impi/4.1.0.024/lib64/libmpi_mt.so section .tbss mismatches non-TLS definition in /opt/gridware/c8000/intel/impi/4.1.0.024/lib64/libmpich.so section .bss
/opt/gridware/c8000/intel/impi/4.1.0.024/lib64/libmpi_mt.so: could not read symbols: Bad value

I'm using -openmp with ifort, and it appears at all compilation and linking commands.

I've since also added -mt_mpi, as well as "-L<mpi_install> -lmpi_mt".

I still get the error.

Here is a shortened version of my final compiler command:

mpiifort -o <executable> -I./Includes -I/opt/gridware/c8000/intel/impi/4.1.0.024/include64 -openmp -mt_mpi -mcmodel=medium -O2 -mieee-fp -align dcommons -extend_source -convert big_endian  <many objects> -L/opt/gridware/c8000/intel/impi/4.1.0.024/lib64 -lmpich -lrt -lpthread -libverbs -libumad -lmpi_mt -L/opt/gridware/lib64 -libcommon

and mpiifort -show =

ifort -ldl -I/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/include -I/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/include -L/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.1 -lmpi -lmpigf -lmpigi -lrt -lpthread

Anybody have any ideas?

[James_T_Intel]

What is your LD_LIBRARY_PATH?

[James_T_Intel]

Also, make certain that any other libraries using MPI are compiled with -mt_mpi.

[tanksnr]

Hi James,

Thanks for having a look.

LD_LIBRARY_PATH = /opt/gridware/applications/PBS/mvapich2/lib:/opt/gridware/compilers/intel/icsxe/composer_xe_2013/lib/intel64:/opt/gridware/compilers/intel/icsxe/itac/8.1.0.024/intel64/slib:/opt/gridware/compilers/intel/icsxe/itac/8.1.0.024/intel64/lib:/opt/gridware/compilers/intel/icsxe/tbb/lib/intel64:/opt/gridware/compilers/intel/icsxe/ipp/lib/intel64:/opt/gridware/compilers/intel/icsxe/mkl/lib/intel64:/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/lib64:/opt/gridware/users/nag/lib64:/opt/gridware/users/nag/lib:/opt/pbs/default/lib/:/opt/gridware/lib64

But that's just for runtime right?

I should've mentioned this first, but the model I use makes use of C for a few routines (primarily the MPI stuff). My mpicc -show gives:

gcc -ldl -I/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/include -L/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.1 -lmpi -lmpigf -lmpigi -lrt -lpthread

And in that makefile I've used:

LIB = -L$(MPI_INST)/lib64 -lmpich -lrt -lmpi_mt

CFLAGS = -O2 -mcmodel=medium -I $(INC)

I've noticed that -lmpi_mt is only used in the final linking process.

Is there an -mt_mpi option for gcc?

I've tried to attach the log of the make here but I think my java is too old.

Can I mail it somewhere?

[tanksnr]

Ok I think I've managed to attach it here.

[James_T_Intel]

It didn't attach, but I don't think that'll be necessary.  You can change -lmpi to -lmpi_mt in the compilation as well.

[tanksnr]

Hi James,

You mean change -lmpi to -lmpi_mt in my mpicc wrapper?

So that mpicc -show will show a gcc compile command with -lmpi_mt INSTEAD of -lmpi?

I didn't compile mpi for our cluster. How do I change what's given by the wrapper without recompiling?

I guess instead of mpicc I could manually use gcc with all the necessary mpicc commands and there change -lmpi to -lmpi_mt.

[TimP]

Adding -lmpi_mt to each use of mpicc should replace the default libmpi with libmpi_mt every time, as James suggested.  It won't change the showme result.

You and your sysadmin don't have the option of rebuilding Intel MPI.  You could make your own copy of the python script in the Intel mpi installation and make a change there, but your sysadmin would be entirely justified in refusing to support you.

If you have an installation of icc and mpiicc, do you have a reason for using gcc?

[tanksnr]

Hi Tim,

My initial attempts (post #3 and #6) with adding -L<impi install>/lib64 -lmpi_mt to the ifort and gcc compiler commands did not solve the problem:

ld: MPIR_Thread: TLS definition in /opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/lib64/libmpi_mt.so section .tbss mismatches non-TLS definition in /opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/lib64/libmpich.so section .bss
/opt/gridware/compilers/intel/icsxe/impi/4.1.0.024/lib64/libmpi_mt.so: could not read symbols: Bad value

Note that this happens for the final linking phase using mpiifort. The mpicc compilation itself seems to go ok (i.e. no errors), although there is no linking done, just ar to create a library. Which is later on linked in the final mpiifort linking that's giving the problem.

Indeed your solution should have worked, since you pointed out that it will replace (or supersede?) the -lmpi shown in the wrapper.

Yes we do have icc on the cluster (13.0.1) but my code specifies gcc in the makefile. Shown below is the makefile for the C codes.

SHELL=/bin/sh

MPI_INST = /opt/gridware/compilers/intel/icsxe/impi/4.1.0.024

LIBNAME = libparlib.a

CC = $(MPI_INST)/bin64/mpicc

LIB = $(MPI_INST)/lib64 -lmpich -lrt -lmpi_mt

INC = $(MPI_INST)/include64

####CFLAGS = -O2 -m32 -I $(INC)
CFLAGS = -O2 -mcmodel=medium -I$(INC) -L$(LIB)

OBJS= par_assoc_buff.o  \
      par_send_noblock.o  \
      par_get_noblock.o  \
      par_wait.o   \
      par_init_put.o \
      par_put_int.o  \
      par_get_int.o  \
      par_put_float.o  \
      par_get_float.o

$(LIBNAME) : $(OBJS)
        $(CC) $(CFLAGS) -c $(?:.o=.c)
        ar -rv $@ $?
        chmod 644 $@
clean:
        rm -rf *.o $(LIBNAME)

[TimP]

I don't know to what extent anyone has been able to make a successful MPI application using the rudimentary MIC gcc, or if mpicc is even tested, as gcc isn't intended to support satisfactory performance.  For example, there is an intentionally non-working mpif90 there.

Your attempt to use libmpi_mt implies that you are trying to make a threaded gcc application.  Were you able to do that in a simple case, not using MPI?

If your point is that your icc is too old to have been tested with a current MIC MPSS, and you may as well use one untested tool as another, that may be true, but MIC programming is difficult enough using tested development tools.

[James_T_Intel]

Why are you using -lmpich?  This links to libmpi, not libmpi_mt.  Try removing that and see if it works.

[tanksnr]

Hi Tim and James,

Thanks for your help.

I could make an omp and non-omp executable without MPI.

I could also make a non-omp executable with MPI.

It was just the omp + MPI I couldn't do.

But James was correct in his last post. I just forgot to remove -lmpich :->.

Didn't even occur to me.

So now I can make an omp + MPI executable!

Edit: The reason for the old library linking in the makefile is that I'm using a relatively old version of a model, which used mpich2.

[James_T_Intel]

Great, I'm glad it's working now!