Undefined reference

[Roberta_G_]

We are running into some undefined references errors when compiling:
mpif90 -o pxa-lnxem64.x sizes.o COMS.o CALL_FORWARD_C.o chord.o dersig.o dertau.o drdnor.o evecl.o evec2.o intsr1.o INVERSE_C.o IO_COEF.o lancos3e.o matr.o max1d2.o mist.o mpi_parameter.o multsend.o pwk.o REC_SORT.o scvd.o sig1.o STRING.o zolgrid.o ITER_GL1CAR_C.o ITER_GL1CAR_C_IN_FR.o ITER_GL1CAR_MC.o ITER_GL1CAR_M.o MAIN.o max2_5ce1.o -lmkl_solver -lmkl_lapack -lmkl_intel_thread -lmkl_em64t -lguide -lpthread -lmpi

/opt/intel/Compiler/11.0/081/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail /opt/intel/Compiler/11.0/081/mkl/lib/em64t/libmkl_lapack.so: undefined reference to `mkl_serv_load_fun'
/opt/intel/Compiler/11.0/081/mkl/lib/em64t/libmkl_lapack.so: undefined reference to `mkl_serv_load_thread_fun'
/opt/intel/Compiler/11.0/081/mkl/lib/em64t/libmkl_intel_thread.so: undefined reference to `mkl_serv_load_lapack_fun'
/opt/intel/Compiler/11.0/081/mkl/lib/em64t/libmkl_intel_thread.so: undefined reference to `mkl_serv_load_lapack_dll'
/opt/intel/Compiler/11.0/081/mkl/lib/em64t/libmkl_lapack.so: undefined reference to `mkl_serv_load_dll' make[1]: *** [pixelinv] Error 1
make[1]: Leaving directory `/people4/jliu17/WORKS/2.5D_FD/Inversion_MPI' make: *** [lnxem64] Error 2

Any thoughts on what might be missing here?

Thanks!!
Roberta

[Gennady_F_Intel]

Roberta,
For linking Intel64 libraries, you need to link lp64 or ilp64 libraries.
Please try to link the following linking lines:
$MKLPATH/libmkl_solver_ilp64.a -Wl,--start-group $MKLPATH/libmkl_intel_ilp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -openmp -lpthread
Where MKLPATH environment variable should point to the directory with MKL libraries
--Gennady

[TimP]

Quoting - Gennady Fedorov (Intel)

Roberta,
For linking Intel64 libraries, you need to link lp64 or ilp64 libraries.
Please try to link the following linking lines:
$MKLPATH/libmkl_solver_ilp64.a -Wl,--start-group $MKLPATH/libmkl_intel_ilp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -openmp -lpthread
Where MKLPATH environment variable should point to the directory with MKL libraries
--Gennady

It should be possible to use the fully dynamic link equivalent of this; as the integrated MKL in the 11.0 compiler is used, it should not then be necessary to specify any MKLPATH.
It looks like the originally quoted library selection may have been based partly on some earlier version of MKL.
The ifort -openmp option already inserts the -lpthread reference, so that is redundant.

[joey_hylton]

Quoting - tim18

It should be possible to use the fully dynamic link equivalent of this; as the integrated MKL in the 11.0 compiler is used, it should not then be necessary to specify any MKLPATH.
It looks like the originally quoted library selection may have been based partly on some earlier version of MKL.
The ifort -openmp option already inserts the -lpthread reference, so that is redundant.

tim18,

I would like to add some comments since this question came from me.

First, I changed the library link a little, and finally the link goes like the following and it works. But the warning is not likely.

mpif90 -o pxa-lnxem64.x sizes.o COMS.o CALL_FORWARD_C.o chord.o dersig.o dertau.o drdnor.o evecl.o evec2.o intsr1.o INVERSE_C.o IO_COEF.o lancos3e.o matr.o max1d2.o mist.o mpi_parameter.o multsend.o pwk.o REC_SORT.o scvd.o sig1.o STRING.o zolgrid.o ITER_GL1CAR_C.o ITER_GL1CAR_C_IN_FR.o ITER_GL1CAR_MC.o ITER_GL1CAR_M.o MAIN.o max2_5ce1.o -lmkl_solver -lmkl_em64t -lguide -lpthread -llibimf -llibm
/opt/intel/Compiler/11.0/081/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail

Second, I have an other code basicly same as this code but without any MPI functions. In other words, it is a sequential version of this code. I used the same link options as those shown in Roberta's post but without the -lmpi .

It works without any problems.

Third, when I submit my job by using qsub, how can I define the thread number, i.e. MKL_NUM_THREADS ? In bash shell, I used "MKL_NUM_THREADS=1; export MKL_NUM_THREADS". It works only for the sequential code, but not the MPI version.

Thanks,

J.L.

[TimP]

Quoting - joey_hylton

mpif90 -o pxa-lnxem64.x sizes.o COMS.o CALL_FORWARD_C.o chord.o dersig.o dertau.o drdnor.o evecl.o evec2.o intsr1.o INVERSE_C.o IO_COEF.o lancos3e.o matr.o max1d2.o mist.o mpi_parameter.o multsend.o pwk.o REC_SORT.o scvd.o sig1.o STRING.o zolgrid.o ITER_GL1CAR_C.o ITER_GL1CAR_C_IN_FR.o ITER_GL1CAR_MC.o ITER_GL1CAR_M.o MAIN.o max2_5ce1.o -lmkl_solver -lmkl_em64t -lguide -lpthread -llibimf -llibm
/opt/intel/Compiler/11.0/081/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail

If you're not attempting to use or IEEE_arithmetic functionality, you probably don't need to be concerned about that warning. It may come from use of the multiple math libraries, which people sometimes do in order to get some functions from each. That also seems outside the scope of this forum. Did you mean -limf -lm ?

You could put the environment variables in your mpi command line by appropriate option such as -genv or other means described in the docs for the mpi of your choice. This would be getting more into the territory of the HPC forum. If you are combining mkl_thread with MPI, you must consider the degree to which your MPI supports it and which special requirements it imposes, as well as whether it requires adherence to the standard about replacing MPI_INIT by MPI_INIT_THREAD.
Intel MPI I_MPI_PIN_DOMAIN has been tested with mkl_thread.

[joey_hylton]

Quoting - tim18

If you're not attempting to use or IEEE_arithmetic functionality, you probably don't need to be concerned about that warning. It may come from use of the multiple math libraries, which people sometimes do in order to get some functions from each. That also seems outside the scope of this forum. Did you mean -limf -lm ?

You could put the environment variables in your mpi command line by appropriate option such as -genv or other means described in the docs for the mpi of your choice. This would be getting more into the territory of the HPC forum. If you are combining mkl_thread with MPI, you must consider the degree to which your MPI supports it and which special requirements it imposes, as well as whether it requires adherence to the standard about replacing MPI_INIT by MPI_INIT_THREAD.
Intel MPI I_MPI_PIN_DOMAIN has been tested with mkl_thread.

tim,

Thanks for the reply.

I don't use in my code. But I am not sure Pardiso uses it or not.

For setting MKL_NUM_THREADS, it didn't work with passing env by MPI. We are searching for other solutions.

You might notice that I try to set MKL_NUM_THREADS=1. In our code, we only use the PARDISO solver, so we tried to use sequential solvers such as MKL_SOLVER_lp64_SEQUENTIAL . However, when we tried to link them, there are always errors complain failing to find some function/subroutines, which is okay if we not try to use sequential ones. We didn't find anything about how to link with sequential libraries in the mklman.pdf.

Do you have any ideas about this?

Linking libraries
MKLLIBSEM64 = -lmkl_solver_lp64_sequential -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_intel_thread -lguide -lpthread -llibimf -llibm


Error message

/opt/intel/Compiler/11.0/081/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
lancos3e.o: In function `lancz_':
lancos3e.f:(.text+0x8e3): undefined reference to `timer_pardiso_'
lancos3e.f:(.text+0x954): undefined reference to `pardiso_'
lancos3e.f:(.text+0xc24): undefined reference to `pardiso_'
lancos3e.f:(.text+0x1080): undefined reference to `pardiso_'
lancos3e.f:(.text+0x1d3b): undefined reference to `pardiso_'
make[1]: *** [pixelinv] Error 1

[VipinKumar_E_Intel]

There is a new MKL link line advisor tool available in MKL KB located here
http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/

--Vipin