compilation ifort mkl_intel_thread impossible

[luckytoyn]

Hi all,

I have recently learned how to make a multithreaded fortran code with openmp and I realized that the most important part of my code is still running on a single CPU.
This comes from two lapack routines I use (dgbtrf and dgbtrs).
I want to use the parallel versions of those two routines (dgbtrf_omp and dgbtrs_omp) but I can't link the mkl library with ifort.

Here is the way I compile :
export INCLUDEMKLPATH=/home/arozel/prog/outils_math_intel/mkl/10.0.1.014/include
export INCLUDEMKLLIB=/home/arozel/prog/outils_math_intel/mkl/10.0.1.014/lib/64
export OMP_NUM_THREADS=2
#ifort -I$INCLUDEMKLPATH -L$INCLUDEMKLLIB -Wl,--start-group -lmkl_intel_thread -lmkl_core -lguide -lmkl_lapack95 -Wl,--end-group -liomp5 -lpthread nonlinbig
.f90
ifort -I$INCLUDEMKLPATH -L$INCLUDEMKLLIB -lmkl_intel_thread -lmkl_core -lguide -lpthread -o par6 nonlinbig.f90


the answer is still the same :
ld: escamotage incompatible /home/arozel/prog/outils_math_intel/mkl/10.0.1.014/lib/64/libmkl_intel_thread.so lors de la recherche de -lmkl_intel_thread
ld: escamotage incompatible /home/arozel/prog/outils_math_intel/mkl/10.0.1.014/lib/64/libmkl_intel_thread.a lors de la recherche de -lmkl_intel_thread
ld: ne peut trouver -lmkl_intel_thread

Yes I know, this is french... Even in french, we are not sure of the meaning...
That seems to mean that ld can't "hide" libmkl_intel_thread.a in a compatible way (I have no idea what this could mean...)
and after that : ld: can't find -lmkl_intel_thread


But if I type
arozel@lst-rozel:~/prog/outils_math_intel/mkl/10.0.1.014/lib/64$ ar -t libmkl_intel_thread.a |grep dgbtrf
it returns
dgbtrf_omp.o

this library exists !!

also :
env |grep LD
LD_LIBRARY_PATH=/usr/lib/petsc/lib/linux-gnu-c-real-opt:/opt/intel/fc/10.1.012/lib:/home/arozel/prog/outils_math_intel/mkl/10.0.1.014/lib/64:/opt/intel/fc/10.1.012/lib
DYLD_LIBRARY_PATH=/opt/intel/fc/10.1.012/lib

I have set the environnement variables on.
I work on a dell precision 690, 64 bits
and ubuntu 6.10


What is wrong with my compilation ? I don't find the answer is the user guide...

Thanks :)

Antoine

[TimP]

I don't think your Dell is an Itanium system, but you show an attempt to link against the Itanium version of MKL. If you are running x86-64, and want the x86-64 code, you must use the intel/fce compiler rather than the intel/fc, and the em64t libraries rather than the /64 (IA-64). If you don't have the time to understand how to install 64-bit compilers under Ubuntu (they do make it difficult), and intend to continue using a 32-bit compiler, you must choose the 32-bit MKL.

"ne peut trouver" appears to be equivalent to the usual "not found," caused by searching for 32-bit libraries in the IA-64 directory.

I admit that the docs are poorly written, even for an English language reader, but it doesn't look like you have examined them. If you decide to switch from i386 to x86-64, you will need the lp64 (or, for 64-bit integer arguments, ilp64) library inside the start-group .. end-group. If using the /32 library, you would instead add the mkl_intel library. You should not attempt to link both libguide and libiomp5, and it would be better to direct ifort to choose one of those. I would expect lapack95 to work when place ahead of start-group.

I don't remember at the moment whether ifort 10.1.012 was the first one which worked reliably with start-group.... end-group; anyway, I would prefer an up to date compiler.

[luckytoyn]

Ok thanks for this reply,
I think I was wrong for a lot of things...

I have just reinstalled the whole system in a 64 architecture (xubuntu 8.04 64 bits), it was installed in 32 bits...
I have installed ifort 64 bits
________________________________
ifort --version
ifort (IFORT) 10.1 20080312
Copyright (C) 1985-2008 Intel Corporation. All rights reserved.
________________________________


I am still trying to link mkl with ifort.
___________________compilation_____________________________________
INCLUDEMKL=/opt/intel/mkl/10.0.1.014/include

MKLPATH=/opt/intel/mkl/10.0.1.014/lib/em64t

ifort nonlinbig.f90 -I$INCLUDEMKL -L$MKLPATH -Wl,--start-group $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end-group -lguide -lpthread
____________________________________________________________________


>______________________answer_______________________________________
/tmp/ifort89q7Fe.o: In function `stream_':
nonlinbig.f90:(.text+0xc47a): undefined reference to `dgbtrf_omp_'
nonlinbig.f90:(.text+0xc63d): undefined reference to `dgbtrs_omp_'
/tmp/ifort89q7Fe.o: In function `invert_lu_':
nonlinbig.f90:(.text+0x16f8b): undefined reference to `dgbtrf_omp_'
nonlinbig.f90:(.text+0x1714e): undefined reference to `dgbtrs_omp_'
_____________________________________________________________________


I just want to link dynamically those two routines which are in libmkl_intel_thread.
I have a xeon dual core cpu.
_________________________________________________________________
*-cpu
product: Intel Xeon CPU 5110 @ 1.60GHz
vendor: Intel Corp.
physical id: 1
bus info: cpu@0
width: 64 bits
_________________________________________________________________

Am I linking with an Itanium version of mkl ?

Thanks a lot for your help.

Antoine

[Todd_R_Intel]

It looks like you are still missing one of the MKL Libraries. Before libmkl_intel_thread.a try adding libmkl_intel_lp64.a. This is the interface library which should catch your call to dgbtrf and dgbtrs and will in turn call the parallel drivers in the threading library. The dgebtrf_omp_ and dgebtrs_omp_ functions are not meant to be called directly and that may be causing some difficulties.
-Todd

[luckytoyn]

In fact, I tried an enourmous sum of combinations...

__________________________________________________
MKLINCLUDE=/opt/intel/mkl/10.0.1.014/include
MKLPATH=/opt/intel/mkl/10.0.1.014/lib/em64t
ifort nonlinbig.f90 -I$INCLUDEMKL -L$MKLPATH -Wl,--start-group $MKLPATH/libmkl_intel_lp64.a $MKLPATH/libmkl_intel_thread.a $MKLPATH/libmkl_core.a -Wl,--end
-group -lguide -lpthread
__________________________________________________
gives the same thing
__________________________________________________
/tmp/ifort2c86XN.o: In function `stream_':
nonlinbig.f90:(.text+0xc47a): undefined reference to `dgbtrf_omp_'
nonlinbig.f90:(.text+0xc63d): undefined reference to `dgbtrs_omp_'
/tmp/ifort2c86XN.o: In function `invert_lu_':
nonlinbig.f90:(.text+0x16f8b): undefined reference to `dgbtrf_omp_'
nonlinbig.f90:(.text+0x1714e): undefined reference to `dgbtrs_omp_'
__________________________________________________

[TimP]

I suspect -Wl,--end-group isn't working when split across lines. If you add the ifort option -# you should see the contents of the linker script which is passed to ld. I think your ifort version is recent enough to have corrected some problems associated with -Wl,--begin-group ....I would put the libraries in order, so that one less pass is required, although it ought to work in your order.

Asyou are linking dynamic OpenMP library, and making the effort toprevent ifort from using its own copy, it seemssomewhat contradictory to link both static and dynamic libraries from MKLPATH, where simply using all dynamic libraries might be easier.

[luckytoyn]

I don't split across lines in my script, only in the post...
If I type this -# (which I didn't knew, very interesting)
I obtain this
__________________
/opt/intel/fce/10.1.015/bin/fortcom
-D__INTEL_COMPILER=1010
-D_MT
-D__ELF__
-D__INTEL_COMPILER_BUILD_DATE=20080312
-D__unix__
-D__unix
-D__linux__
-D__linux
-D__gnu_linux__
-Dunix
-Dlinux
-D__x86_64
-D__x86_64__
-mGLOB_pack_sort_init_list
-I.
-I-L/opt/intel/mkl/10.0.1.014/lib/em64t
-I/opt/intel/fce/10.1.015/include
-I/opt/intel/fce/10.1.015/substitute_headers
-I/usr/lib/gcc/x86_64-linux-gnu/4.2.3/include
-I/usr/local/include
-I/usr/include
-I/usr/lib/gcc/x86_64-linux-gnu/4.2.3/include
"-fp_modspec fp_speculation_FAST"
-O2
-mP1OPT_version=1010
-mGLOB_source_language=GLOB_SOURCE_LANGUAGE_F90
-mGLOB_tune_for_fort
-mGLOB_use_fort_dope_vector
-mP2OPT_static_promotion
-mP1OPT_print_version=FALSE
-mP3OPT_use_mspp_call_convention
-mCG_use_gas_got_workaround=F
-mP2OPT_align_option_used=TRUE
"-mGLOB_options_string=-I -L/opt/intel/mkl/10.0.1.014/lib/em64t -# -Wl,--start-group -Wl,--end-group -lguide -lpthread"
-mGLOB_cxx_limited_range=FALSE
-mP2OPT_eh_nirvana
-mGLOB_diag_file=/tmp/ifortoU8kOL.diag
-mGLOB_as_output_backup_file_name=/tmp/ifortge58Pfas_.s
-mGLOB_machine_model=GLOB_MACHINE_MODEL_EFI2
-mGLOB_fp_speculation=GLOB_FP_SPECULATION_FAST
-mGLOB_extended_instructions=0x8
-mP2OPT_subs_out_of_bound=FALSE
-mGLOB_ansi_alias
-mIPOPT_ninl_user_level=2
-mIPOPT_args_in_regs=0
-mPGOPTI_value_profile_use=T
-mIPOPT_activate
-mIPOPT_lite
-mP2OPT_hlo_level=2
-mP2OPT_hlo
-mPAROPT_par_report=1
-mCG_emit_as_seg_grouping
-mIPOPT_obj_output_file_name=/tmp/ifortoU8kOL.o
"-mGLOB_linker_version=(GNU Binutils for Ubuntu) 2.18.0.20080103"
-mP3OPT_asm_target=P3OPT_ASM_TARGET_GAS
-mGLOB_obj_output_file=/tmp/ifortoU8kOL.o
-mGLOB_source_dialect=GLOB_SOURCE_DIALECT_FORTRAN
-mP1OPT_source_file_name=nonlinbig.f90
nonlinbig.f90

ld
/usr/lib/gcc/x86_64-linux-gnu/4.2.3/../../../../lib64/crt1.o
/usr/lib/gcc/x86_64-linux-gnu/4.2.3/../../../../lib64/crti.o
&nb sp; /usr/lib/gcc/x86_64-linux-gnu/4.2.3/crtbegin.o
--eh-frame-hdr
-dynamic-linker
/lib64/ld-linux-x86-64.so.2
-o
a.out
/opt/intel/fce/10.1.015/lib/for_main.o
/tmp/ifortoU8kOL.o
--start-group
/opt/intel/mkl/10.0.1.014/lib/em64t/libmkl_intel_lp64.a
/opt/intel/mkl/10.0.1.014/lib/em64t/libmkl_intel_thread.a
/opt/intel/mkl/10.0.1.014/lib/em64t/libmkl_core.a
--end-group
-lguide
-lpthread
-L/opt/intel/fce/10.1.015/lib
-L/usr/lib/gcc/x86_64-linux-gnu/4.2.3/
-L/usr/lib/gcc/x86_64-linux-gnu/4.2.3/../../../../lib64
-Bstatic
-lifport
-lifcore
-limf
-lsvml
-Bdynamic
-lm
-Bstatic
-lipgo
-lirc
-Bdynamic
-lc
-lgcc_s
-lgcc
-Bstatic
-lirc_s
-Bdynamic
-ldl
-lc
/usr/lib/gcc/x86_64-linux-gnu/4.2.3/crtend.o
/usr/lib/gcc/x86_64-linux-gnu/4.2.3/../../../../lib64/crtn.o
_______________________________________________

I can see all the libraries I need...
I don't understand when you say that I'm linking with both static and dynamic, what should I write ? I thought I was only linking with dynamic libraries.

[TimP]

You have specified 3 static (.a) MKL libraries.If you are satisfied to use dynamic libraries, rather than using the begin-group ... end-group with the .a libraries, you can simply use-lmkl_intel_lp64 -lmkl_core -lmkl_intel_threadso you don't depend so much on up to date ifort and binutils.

That's another point, your relatively old ubuntu may not have a fully functioning binutils where the begin-group...end-group directives work well. For this reason, my customers simply list the static libraries 3 times in correct order. It's too much to expect us to know how out of date the binutils which come by default with various distros may be, or whether you have run online updates or updated them yourself. I'm somewhat surprised that the Intel compiler bug report form doesn't ask for binutils version.

[Todd_R_Intel]

A simple test program and build script which reproduces this problem, submitted to premier support might help, if this is still a problem.