Undefined MKL symbol when calling from within offloaded region

[mguy44]

Hello,

In an offloaded region of a Fortran90 application, I want to call MKL routines (dgetri/dgetrf) in a sequential way, that is, each thread on the MIC calls these routines with its own data. They aren't multithreaded calls.

I use the Intel® Math Kernel Library Link Line Advisor (v4.4) .

For : Linux / Compiler assisted offload / Intel Fortran / Dynamic / LP64 / Sequential , I get back :

link line :  -L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lm

compiler options :  -I${MKLROOT}/include -offload-attribute-target=mic -offload-option,mic,compiler," -L${MKLROOT}/lib/mic -lmkl_intel_lp64 -lmkl_core -lmkl_sequential"

So I use them, I successfully compile the application and run my code : it crashes on the first call to a MKL routine (here dgetri) with the following error message :

On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/tmp/coi_procs/1/8056/load_lib/ifortoutZ78m5J: undefined symbol: dgetri_"
On the remote process, dlopen() failed. The error message sent back from the sink is /var/volatile/tmp/coi_procs/1/8056/load_lib/ifortoutZ78m5J: undefined symbol: dgetri_
offload error: cannot load library to the device 0 (error code 20)

The environment variables for the compiler, for the MKL are set with the proper scripts, for example

echo $MIC_LD_LIBRARY_PATH 
/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:
/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/mic/coi/device-linux-release/lib:
/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic

(Carriage returns are mine, for readability)

I try the env. var. OFFLOAD_REPORT but I don't get anything else.

So could someone tell me what's going wrong with compile or link step ?

Moreover, is there a place where I can find the signification of error messages received during runtime on the MIC ? For example, in other situations, I got the following message

offload error: cannot start process on the device 0 (error code 22)

but micctrl tells me everything's fine.

Thank you in advance for your advices.

Regards

   Guy.

[Roman_D_Intel1 · Accepted solution]

Hi Guy,

Could you please try adding

-offload-option,mic,ld," -L${MKLROOT}/lib/mic -lmkl_intel_lp64 -lmkl_core -lmkl_sequential"

?

[mguy44]

Hello Roman,

Thank you for your message.

I try adding your option (only for the compilation of this specific routine), I get this new message during the compilation :

ifort: warning #10145: no action performed for file '/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic/libmkl_intel_lp64.a'
ifort: warning #10145: no action performed for file '/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic/libmkl_core.a'
ifort: warning #10145: no action performed for file '/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic/libmkl_intel_thread.a'

At runtime, I encounter the same problem :

On the remote process, dlopen() failed. The error message sent back from the sink is /var/volatile/tmp/coi_procs/1/8088/load_lib/ifortoutolzO8M: undefined symbol: dgetri_
On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/tmp/coi_procs/1/8088/load_lib/ifortoutolzO8M: undefined symbol: dgetri_"
offload error: cannot load library to the device 0 (error code 20)

[Kevin_D_Intel]

The options Roman suggested adding apply to the linking phase and not compilation where you apparently added them. As per the compiler warnings you received, the libraries were ignored for compilation since they don’t apply to that phase. Try adding the options he suggested so they apply to the linking phase.

[mguy44]

Thanks a lot. It works !

I was stupid not to try this.

I go further now ...

Regards,

[Ravi_N_Intel]

If you are using icc to compile and link then you could instead just use -mkl option.

[Kevin_D_Intel]

Glad to hear that worked.

As Ravi suggests, you might be able to simplify life using the -mkl option and placing that in both your compile and link options sets in place of the complicated set of MKL linker options the advisor provided; however, beware that there may be situations where the default MKL library options associated with the -mkl option may not match the advice of the MKL link advisor. For now, trust Roman's advice. He's certainly a better source for advice about this than I.