Compilation shared libraries problem

Pavel_B_ · ‎03-15-2014

Hello everyone,

could someone help me with this issue? I try to compile some OpenMP code, but still wrote this for offload and native compilation:

bar0088@argexpr2:~/Workspace> icc -openmp vec_add.c -o vec_add.out
ld: warning: libimf.so, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)

If i specify the link to the libraries with command -L /opt/intel/composerxe/lib/mic

or

-L /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64

bar0088@argexpr2:~/Workspace> icc -openmp vec_add.c -o vec.out -L /opt/intel/composerxe/lib/mic

ld: skipping incompatible /opt/intel/composerxe/lib/mic/libiomp5.so when searching for -liomp5
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libimf.a when searching for -limf
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libsvml.a when searching for -lsvml
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libirng.a when searching for -lirng
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libipgo.a when searching for -lipgo
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libiomp5.so when searching for -liomp5
ld: skipping incompatible /opt/intel/composerxe/lib/mic/liboffload.so.5 when searching for liboffload.so.5
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libcilkrts.so.5 when searching for libcilkrts.so.5
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libirc.a when searching for -lirc
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libsvml.a when searching for -lsvml
ld: skipping incompatible /opt/intel/composerxe/lib/mic/libirc_s.a when searching for -lirc_s
ld: warning: libimf.so, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/liboffload.so.5, not found (try using -rpath or -rpath-link)

I have no idea how to fix it. What is wrong?

Thanks

Kevin_D_Intel · ‎03-17-2014

Did you source the compilervars.sh/csh script?

If so check to ensure something has not modified the LD_LIBRARY_PATH after using the setup script. It must at least include /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64 to avoid that error. The compilervars script also adds additional paths depending on what other Composer components may be installed.

Pavel_B_ · ‎03-17-2014

Yes, I used script command /opt/intel/composerxe/bin/compilervars.sh intel64 and after compilation shows that 4 warnings... output file need some libraries, but some are not found. When I try run it, fails because it's damage.

bar0088@argexpr2:~/Workspace> ldd Matrix_offload.out                                                   linux-vdso.so.1 => (0x00007ffff45ff000)
        libmkl_intel_lp64.so => not found
        libmkl_intel_thread.so => not found
        libmkl_core.so => not found
        libiomp5.so => not found
        libm.so.6 => /lib64/libm.so.6 (0x00007faf8ddfc000)
        liboffload.so.5 => not found
        libcilkrts.so.5 => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007faf8daf1000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007faf8d8db000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faf8d6be000)
        libc.so.6 => /lib64/libc.so.6 (0x00007faf8d345000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007faf8d141000)
        /lib64/ld-linux-x86-64.so.2 (0x00007faf8e0ae000)

If i linked these 4 libraries (libimf.so, libsvml.so, libirng.so, libintlc.so) manually from /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64, it writes incompatible libraries.

How I check that LD_LIBRARY_PATH include /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64 ?

Thank you

Kevin_D_Intel · ‎03-17-2014

This same variable can lead to run-time failures also like those you showed and there are also other important env-variables like, MIC_LD_LIBRARY_PATH.

To view the environment variable setting, enter the shell command:

echo $LD_LIBRARY_PATH

On my system, it contains:

$ echo $LD_LIBRARY_PATH
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:
/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:
/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:
/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:
/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.4

You might also wish to check MIC_LD_LIBRARY_PATH.

$ echo $MIC_LD_LIBRARY_PATH
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:
/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/mic/coi/device-linux-release/lib:
/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic

The errors you are experiencing suggests something is perhaps altering the values of these env-variables after the compilervars script runs or something is malfunctioning with the script itself.

When you the compilervars.sh script, did you use it with the "source" command like this:

source /opt/intel/composerxe/bin/compilervars.sh intel64

Pavel_B_ · ‎03-17-2014

//edit: Added twice..

Pavel_B_ · ‎03-17-2014

That was the problem, I used it without source command.

Now it compiles well, but when I try run some offload code it wrote line 1:ELF: not found. Here is simple code, that I tried and commands from bash I used (I tried more offload examples with the same error when I ran it).

bar0088@argexpr2:~/Workspace> source /opt/intel/composerxe/bin/compilervars.sh intel64
bar0088@argexpr2:~/Workspace> source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64
bar0088@argexpr2:~> cd /home/fei/bar0088/Workspace/                             
bar0088@argexpr2:~/Workspace> icpc -mkl -openmp -O3 -Wno-unknown-pragmas -vec-report2 pi.c -o pi.out
pi.c(8): (col. 1) remark: *MIC* LOOP WAS VECTORIZED
bar0088@argexpr2:~/Workspace> scp pi.out mic0:/tmp                              
pi.out                                        100%  227KB 227.3KB/s   00:00
bar0088@argexpr2:~/Workspace> ssh mic0
[bar0088@cct407-6-mic0 bar0088]$ cd /tmp
[bar0088@cct407-6-mic0 /tmp]$ ./pi.out
./pi.out: line 1:ELF: not found
./pi.out: line 1: syntax error: unexpected word (expecting ")")
[bar0088@cct407-6-mic0 /tmp]$

And simple offload example to compute pi:

#include <iostream>

int main(int argc, char* argv[]) {
   const int niter = 100000;
   double result = 0;

   #pragma offload target(mic) 
   for (int i = 0; i < niter; ++i) {
      const double t = (i + 0.5) / niter; result += 4.0 / (t * t + 1.0);
   } 
   result /= niter; 
   std::cout << "Pi ~ " << result << '\n';
}

Kevin_D_Intel · ‎03-17-2014

compilervars.sh calls mklvars.sh on your behalf when it detects MKL is installed. Sourcing mklvarls again as you showed isn't too harmful but it adds duplicate entries into env-variables so you can skip that step.

The ELF runtime failure resulted from not compiling with -mmic to produce a native Intel® Xeon Phi™ executable; however, that does not appear to be what you want based on the program you showed. The program shown will offload the for loop to the card (if available) for you. You just compile the program as you showed and then execute pi.out on the host.

If you are interested in building/running codes natively and not using the language extensions for offload, then refer to the article Building a Native Application for Intel® Xeon Phi™ Coprocessors.

Pavel_B_ · ‎03-17-2014

I write thesis about Xeon Phi in which I deal with both, native and offload mode. Native mode I understand and it's functional, but with offload mode I have nothing but problems :/

I have more complicated application that I want run in offload and compare it with native, that is my intention. That Pi was only test sample. When I run it from host, it's working, stupid mistake to run it of bash of card.

Many thanks for help!

Kevin_D_Intel · ‎03-17-2014

I'm sorry you are having problems with offload. Maybe take a look the samples under:

/opt/intel/composer_xe_2013_sp1.2.144/Samples/en_US/C++/mic_samples

Those include a makefile that might make clearer what I'm trying to describe about how you build/run offload-enabled programs from the host. I don't recall seeing ELF failures on the host when running offload-enabled program so if that's what you are also seeing then I'm not sure what is missing to cause that offhand. I'd have to experiment to see whether I could re-create that.

Pavel_B_ · ‎03-17-2014

Thanks, I look on that. Now I tried run that pi sample in offload and gives me error again.

bar0088@argexpr2:~/Workspace> source /opt/intel/composerxe/bin/compilervars.sh intel64
bar0088@argexpr2:~/Workspace> export OFFLOAD_REPORT=3
bar0088@argexpr2:~/Workspace> ./pi_offload.out
offload error: cannot start process on the device 0 (error code 9)
[Offload] [HOST]  [State]   Unregister data tables

Kevin_D_Intel · ‎03-17-2014

As per /opt/mpss/3.1.2/sysroots/k1om-mpss-linux/usr/include/intel-coi/common/COIResult_common.h, that appears to be the result of a COI timeout:

COI_TIME_OUT_REACHED, ///< The specified time out caused the
///< function to abort.

Did you happen to try running any of the MIC offload samples?

From your earlier posts it seems your card is booted. It might not show any issues, but try executing: miccheck

If that does not report any issues, then reboot the card and re-run the offload program.

Pavel_B_ · ‎03-17-2014

Yes, I tried these samples, all 14 passed but in the end it wrotes again Unregister data tables, is it normal? Below is miccheck which failed.

Samples complete
[Offload] [MIC 0] [State] Unregister data tables
[Offload] [HOST] [State] Unregister data tables

bar0088@argexpr2:~/Workspace> /opt/intel/mic/bin/miccheck

miccheck 2.1.6720-15, created 11:31:20 Jun 21 2013
Copyright 2011-2013 Intel Corporation  All rights reserved

Test 1 Ensure installation matches manifest : OK
Test 2 Ensure host driver is loaded         : OK
Test 3 Ensure driver matches manifest       : OK
Test 4 Detect all listed devices            : OK
MIC 0 Test 1 Find the device                       : OK
MIC 0 Test 2 Check the POST code via PCI           : OK
sh: micctrl: command not found
MIC 0 Test 2> Shell command 'micctrl -s mic0' returned error (non-zero)
MIC 0 Test 3 Connect to the device                 : SKIPPED
MIC 0 Test 3> Prerequisite 'Ensure the device is online' failed:
MIC 0 Test 3>  The device is not online
MIC 0 Test 4 Check for normal mode                 : SKIPPED
MIC 0 Test 4> Prerequisite 'Ensure the device is online' failed:
MIC 0 Test 4>  The device is not online
MIC 0 Test 5 Check the POST code via SCIF          : SKIPPED
MIC 0 Test 5> Prerequisite 'Ensure the device is online' failed:
MIC 0 Test 5>  The device is not online
MIC 0 Test 6 Send data to the device               : SKIPPED
MIC 0 Test 6> Prerequisite 'Check for normal mode' failed:
MIC 0 Test 6>  The device is not in normal mode
MIC 0 Test 7 Compare the PCI configuration         : OK
MIC 0 Test 8 Ensure Flash version matches manifest : SKIPPED
MIC 0 Test 8> Prerequisite 'Check for normal mode' failed:
MIC 0 Test 8>  The device is not in normal mode
Status: Test failed

Kevin_D_Intel · ‎03-17-2014

I assume the samples reported Passed? If so, then it is puzzling why the pi test appears to experience the COI timeout.

The miccheck errors you experienced occur when PATH does not include: /usr/sbin

Try executing these commands:

export PATH=$PATH:/usr/sbin
/opt/intel/mic/bin/miccheck
/opt/intel/mic/bin/micinfo

Paste the output into a reply.

Pavel_B_ · ‎03-18-2014

Here is miccheck and micinfo. It seems to be OK. So what may cause that COI timeout?

bar0088@argexpr2:~> export PATH=$PATH:/usr/sbin
bar0088@argexpr2:~> /opt/intel/mic/bin/miccheck

miccheck 2.1.6720-15, created 11:31:20 Jun 21 2013
Copyright 2011-2013 Intel Corporation  All rights reserved

Test 1 Ensure installation matches manifest : OK
Test 2 Ensure host driver is loaded         : OK
Test 3 Ensure driver matches manifest       : OK
Test 4 Detect all listed devices            : OK
MIC 0 Test 1 Find the device                       : OK
MIC 0 Test 2 Check the POST code via PCI           : OK
MIC 0 Test 3 Connect to the device                 : OK
MIC 0 Test 4 Check for normal mode                 : OK
MIC 0 Test 5 Check the POST code via SCIF          : OK
MIC 0 Test 6 Send data to the device               : OK
MIC 0 Test 7 Compare the PCI configuration         : OK
MIC 0 Test 8 Ensure Flash version matches manifest : OK
Status: OK.


bar0088@argexpr2:~> /opt/intel/mic/bin/micinfo
MicInfo Utility Log

Created Tue Mar 18 08:20:20 2014


        System Info
                HOST OS                 : Linux
                OS Version              : 3.0.80-0.7-default
                Driver Version          : 6720-15
                MPSS Version            : 2.1.6720-15
                Host Physical Memory    : 1025538 MB

Device No: 0, Device Name: mic0

        Version
                Flash Version            : 2.1.03.0386
                SMC Firmware Version     : 1.15.4830
                SMC Boot Loader Version  : 1.8.4326
                uOS Version              : 2.6.38.8-g2593b11
                Device Serial Number     : ADKC32401120

        Board
                Vendor ID                : 0x8086
                Device ID                : 0x2250
                Subsystem ID             : 0x2500
                Coprocessor Stepping ID  : 3
                PCIe Width               : Insufficient Privileges
                PCIe Speed               : Insufficient Privileges
                PCIe Max payload size    : Insufficient Privileges
                PCIe Max read req size   : Insufficient Privileges
                Coprocessor Model        : 0x01
                Coprocessor Model Ext    : 0x00
                Coprocessor Type         : 0x00
                Coprocessor Family       : 0x0b
                Coprocessor Family Ext   : 0x00
                Coprocessor Stepping     : B1
                Board SKU                : INVALID SKU
                ECC Mode                 : Enabled
                SMC HW Revision          : Product 225W Passive CS

        Cores
                Total No of Active Cores : 60
                Voltage                  : 997000 uV
                Frequency                : 1052631 kHz

        Thermal
                Fan Speed Control        : N/A
                Fan RPM                  : N/A
                Fan PWM                  : N/A
                Die Temp                 : 49 C

        GDDR
                GDDR Vendor              : Elpida
                GDDR Version             : 0x1
                GDDR Density             : 2048 Mb
                GDDR Size                : 7936 MB
                GDDR Technology          : GDDR5
                GDDR Speed               : 5.000000 GT/s
                GDDR Frequency           : 2500000 kHz
                GDDR Voltage             : 1501000 uV
bar0088@argexpr2:~>

Kevin_D_Intel · ‎03-18-2014

I overlooked your earlier comment that the samples reported "Passed". I'm perplexed how the samples pass but the simple pi test suffers the offload failure, seemingly due to a COI timeout.

I have never been able to synthesize the (error code 9) and I do not have enough knowledge of the inner-workings to know what can cause that specifically. Apparently something within the COI-layer determined a timeout occurred with the attempted offload between the host and coprocessor. The offload runtime library does not establish any timeouts itself based on what offload compiler Developers have said in the past regarding this error.

Will you please try running the ./pi_offload.out again to see whether the offload failure repeats?

If so, (as root) cycle MPSS with the command: service mpss restart
And then try re-running the pi executable.

Pavel_B_ · ‎03-20-2014

I don't know how, but command service mpss restart was effective, now it works fine, even offload. I do not have administrator rights because it took a while.

Kevin_D_Intel · ‎03-20-2014

Glad to hear that. Just guessing, the reboot either restarted a damaged process on the card involved in managing offload execution (unlikely since the samples ran ok) or (more likely) removed remnants from the card's filesystem image that were created from earlier failed (ELF errors) or successful attempts running the pi test natively. Whatever it was it was somehow unique to the pi executable and affected running it only in offload but it did not affect the samples running in offload. I know that well before the /tmp/coi_procs structure was deployed it was possible to pollute /tmp with similar named files as host offload enabled executables that would break offload execution; however, I have never seen that again since the coi_procs has been in place.

When working in native mode, choose the working directory carefully and where files are transferred to the card and remove files when the native testing is complete. micnativeloadex is a very handy utility also.