offload error: unexpected embedded target binary type, ...

[Hong-Hyun_P_]

Hello,
Could someone help me with my software installation issue with Xeon Phi? I'm trying to compile my program on the host for future offload developments. In the case I don't include any offload functions in my codes, my program as well as some external libraries (Arpack, Petsc, Libmesh, etc.) are compiled with Intel MPI & MKL and running well on the host. However, if I put any testing offload codes, then it shows long messages during compilation and linking as shown below. An executable is created anyways but it's not running.
By the way, I don't see any problem when I try a sample offload example (/apps/intel/composer_x3_2015.2.164/Samples/en_US/C++/mic_samples/LEO_tutorial/tbo_sort.c) with the same system environment.
Could you please see the following messages and give any advices?
Thank you,
Hong

==== I don't see any 'skipping inconpatible...' messages If I don't put any offload codes ====
[my_account@host_system]$ make
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libiomp5.so when searching for -liomp5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libiomp5.a when searching for -liomp5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libimf.a when searching for -limf
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libsvml.a when searching for -lsvml
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libirng.a when searching for -lirng
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libipgo.a when searching for -lipgo
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libdecimal.a when searching for -ldecimal
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libiomp5.so when searching for -liomp5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libiomp5.a when searching for -liomp5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/liboffload.so.5 when searching for liboffload.so.5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libiomp5.so when searching for -liomp5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libiomp5.a when searching for -liomp5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libcilkrts.so.5 when searching for libcilkrts.so.5
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libirc.a when searching for -lirc
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libsvml.a when searching for -lsvml
x86_64-k1om-linux-ld: skipping incompatible /apps/intel/lib/intel64/libirc_s.a when searching for -lirc_s
x86_64-k1om-linux-ld: warning: libimf.so, needed by /tmp/icpcJ9JZwzMIC.a, not found (try using -rpath or -rpath-link)
x86_64-k1om-linux-ld: warning: libsvml.so, needed by /tmp/icpcJ9JZwzMIC.a, not found (try using -rpath or -rpath-link)
x86_64-k1om-linux-ld: warning: libirng.so, needed by /tmp/icpcJ9JZwzMIC.a, not found (try using -rpath or -rpath-link)
x86_64-k1om-linux-ld: warning: libintlc.so.5, needed by /tmp/icpcJ9JZwzMIC.a, not found (try using -rpath or -rpath-link)
[my_account@host_system]$ ./my_program
offload error: unexpected embedded target binary type, expected either an executable or shared library
[my_account@host_system]$
=============

[Frances_R_Intel]

You list a number of external libraries that your program uses. These libraries must be compiled for the coprocessor as well as the host. You might want to look at the documentation, particularly the User and Reference Guide for the Intel C++ Compiler. The section on Programming for Intel MIC Architecture has subsections on creating and using offload libraries. That section from the documentation for the 15.0 compiler is at https://software.intel.com/en-us/node/522481. If your libraries are using configure scripts to set up their build, you might want to look at https://software.intel.com/en-us/articles/compiling-open-source-for-intel-xeon-phi-coprocessors for advice on how to do that.

As to the error messages you are seeing - what they are saying is that they can find the host version of libiomp5.so but not the coprocessor version. By default, the compilers and the libraries that come with them are installed in /opt/intel and the script /opt/intel/composerxe/bin/compilervars.sh sets the environment variables needed to build and run your code. There are MIC versions of some of the environment variables, like MIC_LD_LIBRARY_PATH, so that you can set the variables differently for the host and coprocessor. I notice that you have your compilers installed in /apps/intel. It is possible that either the lib/mic files did not find there way to new location or that the special MIC environment variables didn't get modified for the new location or, most likely since the sample code works, there is some error on your command line.

Hope that helps you figure out what is going wrong.

[Hong-Hyun_P_]

Hello Frances and thank you for your kind explanations. I've read some of the documents you suggested but couldn't fix my issue yet.I think I need to explain more about my problem.
I'm pursuing the MPI host-only model explained at the section 6.2.3 of (http://www.prace-ri.eu/best-practice-guide-intel-xeon-phi-html/#section-offload). For that, I'm going to use offloading pragmas only, which as far as I understand does not require any special compilation procedures because the code blocks for MIC will be automatically generated together with the code blocks for the host. (section 4. of the above document). Also, I'm sure that any of the external libraries will not be offloaded in my program. Actually, for my test I set MIC_LD_LIBRARY_PATH and MIC_ENV_PREFIX and put few lines of very simple offloading and openmp pragmas into the main function, which results in the compiling and runtime problems I reported before.
In summary, I thought I just need to compile my program only on the host with -openmp option and run it on the host, too, because I want the host-only MPI offloading model. Please correct me if my understanding is not correct.

[JJK]

check the compiler and loader flags (CFLAGS and LDFLAGS) in your Makefile - is there any reference to "-mmic"  ? in that case the code is compiled for Phi native mode and you would end up with the error messages you mention.

[Hong-Hyun_P_]

I'm sure that no '-mmic' option is used anywhere in my environment. Also, as I mentioned, my program is compiled and running well on the host if I don't put few lines of simple testing offload & openmp pragmas into the main function. The insertion is the only change for the error. So, I don't think '-mmic' option is the cause of the problem. Are there other possibilities? 

 

[JJK]

Hmmm , that's odd, as it is the mic loader (x86_64-k1om-linux-ld) that is called. Are you setting the "LD" env var at all? is there a way to see a verbose log of the make process?

[Frances_R_Intel]

Could you tell me what you are setting MIC_ENV_PREFIX and MIC_LD_LIBRARY_PATH to? The probably is most likely in there.

[Hong-Hyun_P_]

I'm using Intel MPI compilers for my program, and the the settings are
LD_LIBRARY_PATH = /apps/intel/composer_xe_2015/lib/intel64:apps/intel/impi/5.0.3.048/intel64/lib:usr/local/lib64:/usr/lib64
MIC_LD_LIBRARY_PATH = /apps/intel/compser_xe_2015/lib/mic:/apps/intel/impi/5.0.3.048/mic/lib
MIC_ENV_PREFIX=MIC
Again, as I mentioned, using the same environment, I could compile and run the sample offload example. That's frustrating me...
 

[Frances_R_Intel]

There are a number of libraries missing from the list. Please try running

source /apps/intel/composerxe/bin/compilervars.sh intel64

Make sure that the directory names show up as /apps/intel since that is the location you are using.

Could you also share what your makefile looks like?

[Hong-Hyun_P_]

Hello,
I see that LD_LIBRARY_PATH is updated after I ran "source /apps/intel/composerxe/bin/compilervars.sh intel64" and/or "source /apps/intel/impi_latest/intel64//bin/mpivars.sh intel64", but there's no difference in the error messages.
Please see attached a part of my Makefile where I put some 'echo's to print out LD_LIBRARY and CXXFLAGS etc., and compile logs w/ and w/o offload codes. You can also see that sample offload tests (a.out, b.out) are running well with the same environment. Thank you so much.

[JJK]

thanks for posting the makefile output; I am no MPI expert here but I suspect that the mpiicpc compiler tries to build a MIC version of the code  (or a MIC offloading module of the code) and then fails to link in the right libraries. Your makefile uses "-Wl,-rpath,/home/honghyun.p/Quatom_mic/src/libs/ARPAC"  and I've seen the Intel linker/loader choke on stuff like that.

What you can try to do is to add two flags to the mpiicpc call: "-show" and "-echo" : this will either show or echo the scripts that the Intel MPI compiler is using under the hood to generate the binaries. It might shed more light on what is going on .

[Hong-Hyun_P_]

Hello. Attached is the messages from 'mpiicpc -show' and 'mpicpc -echo'. I have no idea what those mean...

[Frances_R_Intel · Accepted solution]

There is offloaded code in libQuatom.so, isn't there? Try adding '-qoffload' to the command line when you make the .so file and when you make the executable. Let's see if we can force the loader to realize it needs the coprocessor libraries as well as the host ones.

Other points -

If you make a call to a library from within an offload section of code, then you need a version of that library containing code compiled for the coprocessor. Given that none of the error messages refers to your local libraries, you are probably right when you say none of those libraries are being used on the coprocessor. But like Jan Just K., I worry about all the path statements on your loader line especially the -rpath. One of the reasons for having both regular LD_LIBRARY_PATH and LIBRARY_PATH environment variables as well as MIC versions is that you can sort out the different paths and make sure the libraries you need all get copied over to the coprocessor. If adding -qoffload doesn't help, could you try cleaning up the loader commands, moving the library locations to LD_LIBRARY_PATH, LIBRARY_PATH, MIC_LD_LIBRARY_PATH or MIC_LIBRARY_PATH, as appropriate and removing the rpath?

If you are using the offload programming style, then you cannot make calls to MPI functions from the offload sections. So you only need the host's MPI library for this code.

[Hong-Hyun_P_]

Dear all,
Adding '-qoffload' didn't work but copying rpath to LD_LIBRARY_PATH and removing the rpath WORKS! I'm not sure but just guess that it's related to the Intel compiler behavior explained in (https://www.osc.edu/supercomputing/computing/ruby/phi_compiling_guide) as "By default, all environment variables defined on he host are replicated to the coprocessors execution environment when an offload occurs.", and it seems that rpath is not copied to coprocessors. Thank you so much for your all the advices.
 

[Frances_R_Intel]

When you run an offload code, the libraries you need are found using the MIC_LD_LIBRARY_PATH variable and copied to a directory on the coprocessor created explicitly for running the offload code. So on the coprocessor, the path you need will not be the same as on the host. As far as the statement from https://www.osc.edu/supercomputing/computing/ruby/phi_compiling_guide, it is true but with a caveat - "except when there are coprocessor specific variables." Sorry about the confusion.