Error in static linking to MKL on Linux x64 platform

[lee_icc]

Dear All,

I tried to compile my C code with Intel C/C++ compiler 13.1.1 and Intel MKL on scientific linux 6.4 x64 platform. When linked dynamically, it worked fine without any error or warnings. When linked with -static, it reported a error:

[sl@localhost test]$ icc -O3 -ip -mkl=parallel -openmp -static mcmc_demo.c
ld: cannot find -lm

Following the webpage http://software.intel.com/en-us/articles/error-ld-cannot-find-lm/, I tested the new compile options, and it reported many error about undefined reference:

[sl@localhost test]$ icc -O3 -ip -mkl=parallel -openmp -static mcmc_demo.c -L/usr/lib/x86_64-redhat-linux5E/lib64/
/tmp/iccondg4F.o: In function `main':
mcmc_demo.c:(.text+0x71): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x100): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x174): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x230): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x43a): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x5ed): undefined reference to `vdLn'
mcmc_demo.c:(.text+0xd00): undefined reference to `vslNewStream'
mcmc_demo.c:(.text+0xd28): undefined reference to `vdRngUniform'
mcmc_demo.c:(.text+0xd42): undefined reference to `vdLn'
mcmc_demo.c:(.text+0xe15): undefined reference to `vdRngGaussian'
mcmc_demo.c:(.text+0xeac): undefined reference to `vdLn'
mcmc_demo.c:(.text+0x1561): undefined reference to `cblas_dcopy'
mcmc_demo.c:(.text+0x159f): undefined reference to `cblas_dcopy'
mcmc_demo.c:(.text+0x1607): undefined reference to `cblas_dcopy'
mcmc_demo.c:(.text+0x1645): undefined reference to `cblas_dcopy'
mcmc_demo.c:(.text+0x16b1): undefined reference to `cblas_dcopy'
mcmc_demo.c:(.text+0x175f): undefined reference to `vslDeleteStream'
/tmp/iccondg4F.o: In function `asm2d':
mcmc_demo.c:(.text+0x8184): undefined reference to `cblas_dgemv'
/tmp/iccondg4F.o: In function `logpriorfun':
mcmc_demo.c:(.text+0x81c3): undefined reference to `vdLn'
/tmp/iccondg4F.o: In function `proprnd':
mcmc_demo.c:(.text+0x8470): undefined reference to `vdRngGaussian'

Does anyone have idea that how to resolve this issue? Thanks in advance.

Lee

[SergeyKostrov]

There is an active thread already and please take a look at: Forum Topic: Error while using -static and -L/usr/lib/x86_64-redhat-linux5E/lib64/ on Linux x64 platform Web-link: http://software.intel.com/en-us/forums/topic/393889

[Ying_H_Intel]

Hi Sergey,  thanks for the message.

Hi Lee,

I have same issue reported before and get fixed in MKL update 4 . So could you please

1. use mkl advisor and try explictly link MKL

-Wl,--start-group  $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_intel_thread.a $(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm

http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/

2. use MKL 11.0 update 4.

3. could you please give me the result of   your command line  -#

Thanks

Ying

[lee_icc]

Hi Ying, Following your advice, I tested several combination of static linking options (-static or -static-intel). Using option -static leads to a 'ld: cannot find -lm' error as shown previously. For others: 1 -static-intel: [sl@localhost test]$ icc -O3 -ip -xHost -openmp -static-intel mcmc_demo.c icc: warning #10237: -lcilkrts linked in dynamically, static library not available /tmp/iccuSIkoV.o: In function `main': mcmc_demo.c:(.text+0x5f9): undefined reference to `vdLn' ... When explictly linking MKL, it was different: 2 -static with -Wl,... [sl@localhost test]$ icc -O3 -ip -openmp -static mcmc_demo.c -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm ld: cannot find -lpthread and finally: 3 -static-intel with -Wl... [sl@localhost test]$ icc -O3 -ip -openmp -static-intel mcmc_demo.c -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm icc: warning #10237: -lcilkrts linked in dynamically, static library not available Only the last compile options built the executalbe succesfully without any error. Actually, I was confused with these static linking options.

[lee_icc]

After several test, I also found that using the option '-L/usr/lib/x86_64-redhat-linux5E/lib64/' resolved the error 'ld: cannot find -lpthread' while leading to undefined reference to `__isoc99_fscanf'. Without this option, it reported 'ld: cannot find -lpthread' again without any undefined reference to `__isoc99_fscanf'. Detailed logs are as follows:

[sl@localhost test]$ icc -O3 -ip -openmp -static mcmc_demo.c -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm
ld: cannot find -lpthread

[sl@localhost test]$ icc -O3 -ip -openmp -static mcmc_demo.c -L/usr/lib/x86_64-redhat-linux5E/lib64/ -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.a $MKLROOT/lib/intel64/libmkl_intel_thread.a $MKLROOT/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm
/tmp/iccZTzhHD.o: In function `main':
mcmc_demo.c:(.text+0x71): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x100): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x174): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x230): undefined reference to `__isoc99_fscanf'
mcmc_demo.c:(.text+0x43a): undefined reference to `__isoc99_fscanf'

[Ying_H_Intel]

Hi Lee_icc,

I think your test are very close the truth.  Just quick share what i think

There are 3 problems here

1) the order of option in command line do matters  as the linux linker only veiw the library, obj etc one times to resolve reference.

so

$icc -O3 -mkl mcmc_demo.c  have unresoved mkl symbol.

and $icc -o3 mcmc_demo.c  -mkl  or the explict mkl library link (as you did) solve the mkl symbols problem.

2) about the static and static-intel

as i understand, static will ask static library, include glibc static library and pthread library, like -lm, -lpthread.

static-intel will ask intel static library.  and allow other system library are be dynamic.

In general, MKL follow linux (discourage static library),  hope the system library be dynamic .

3) the __isoc99_fscanf  should be from some library of glibc.   I double glibc dynamic library can resolve the symbol, but the static glibc library may not on your machine.  

So according to the condtion,  you may try

1) drop  the -static  in your last command (sorry, i haven't copied the command as if i did,  the IE hang ) . Thus MKL static library  and system dynamic library are linked, which is actually recommended model.

2) if you really need to all libraries include glibc library are static, then you may find which static library include the symbol __isoc99_fscanf. then you will resolve the problem.

Best Regards,

Ying

[Ying_H_Intel]

and the 3  static-intel with -Wl  you try last last time.  also are linking static MKL library (and static intel library except the libcilkrts)  and dynamic system library,  so it is not problem too.

Best Regards,

Ying

[lee_icc]

Thank you very much for your advice and knowledge, Ying.

[SergeyKostrov]

>>...3) the __isoc99_fscanf should be from some library of glibc... I searched for that substring in a folder where a GCC-like C++ compiler installed and I did not see any references. It looks like I need to update my version of the compiler.

[lee_icc]

Hi Ying, you are right. The order of option in command line indeed affect the compilation in certain conditions. I did more tests on the compilation options. When linking dynamically, both the '$ icc -mkl mcmc_demo.c' and '$icc mcmc_demo.c -mkl' can compile correctly. While linking statically, the situation was different. For the option -static which ask static libraries (including system library and Intel library), using -static resulted in the error of not finding the static system library ('ld: cannot find -lm') wherever -mkl before or after mcmc_demo.c. When using -static together with -L/usr/lib/x86_64-redhat-linux5E/lib64/, the compiler found the static system library but reported undefined reference of __isoc99_fscanf and MKL symbols. Only placing -mkl after the mcmc_demo.c can resolve the MKL symbols reference issue. Therefore, as I understand from the tests above, under the static linking condition with -static, compiler search static system libraries firstly, and then Intel libraries. When static system libraries is not available, the compilation terminate without searching static Intel libraries and report the issue, and consequently the order of -mkl have no effect. On the other side, when compiler find the static system libraries (with -L/usr/lib/x86_64-redhat-linux5E/lib64/), it try to find static Intel libraries further. At this time, -mkl have to be placed after mcmc_demo.c, or using the explict MKL linking can resolve the MKL symbols reference issue. At last, for the option -static-intel which ask static Intel library, again, -mkl have to be placed after mcmc_demo.c, or using explict MKL linking to give correct compilation. In all, when using the option -static or -static-intel and system libraries are linked correctly (dynamic or static), -mkl have to be placed after the src filename to complete the correct compilation.

[lee_icc]

To solve the __isoc99_fscanf reference issue, I installed the glibc-static.x86_64 package by following http://software.intel.com/en-us/articles/error-ld-cannot-find-lm/. It looks like the __isoc99_fscanf reference issue was resolved: [sl@localhost test]$ icc -O3 -ip -openmp -static mcmc_demo.c -mkl /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libpthread.a(libpthread.o): In function `sem_open': (.text+0x764d): warning: the use of `mktemp' is dangerous, better use `mkstemp'

[Ying_H_Intel]

Hi lee_icc,

Thank you for sharing. right, the __iso9_fscanf, should be in static glibc library. 

Regarding the last warning in libpthread.a, it doesn't matter,  Same reason that  the static system library is not encourage to used too. 

Best Regards,

Ying