How do I link Intel MKL and libdl with gold linker?

[Andrew_T_]

I'm having a problem linking Intel MKL and libdl using the gold linker on CentOS:

When I run this script:

#!/bin/bash

MKL_INC=$MKL_INSTALL_DIR/include
MKL_LIB=$MKL_INSTALL_DIR/lib

. /opt/rh/devtoolset-6/enable

cat > t.c << end_mkltest

#include <dlfcn.h>
#include "mkl_service.h"

int main() {
    dlerror();              /* use libdl */
    mkl_set_num_threads(1); /* use mkl   */
}

end_mkltest

gcc -I$MKL_INC -c t.c -o t.o
gcc -L$MKL_LIB -fuse-ld=gold t.o -lmkl_rt -ldl

I get:

libmkl_rt.so: error: undefined reference to 'calloc'
libmkl_rt.so: error: undefined reference to 'realloc'
libmkl_rt.so: error: undefined reference to 'malloc'
libmkl_rt.so: error: undefined reference to 'free'

We're using:

CentOS 7.3
devtoolset-6
mkl-2017.2.174.tar.bz2

Any ideas?

[Gennady_F_Intel]

Andrew, please try to change the MKL_LIB like as follows: MKL_LIB=$MKL_INSTALL_DIR/lib/intel64

[Andrew_T_]

The way we reposition the libraries this would have no effect, but just in case:

1. I tried appending intel64, and it did not fix it (we still get an identical error).

2. I also tried with a vanilla install of l_mkl_2017.3.196.tgz, without any processing and it did not fix it (we still get an identical error).

Any other ideas?

[Ying_H_Intel]

Hi Andrew,

I did some investigation in internet.  it seems because that gold linker

https://stackoverflow.com/questions/3476093/replacing-ld-with-gold-any-experience

At the moment it is compiling bigger projects on Ubuntu 10.04. Here you can install and integrate it easily with the binutils-gold package (if you remove that package, you get your old ld). Gcc will automatically use gold then.

Some experiences:

• gold doesn't search in /usr/local/lib
• gold doesn't assume libs like pthread or rt, had to add them by hand

While undefined reference to 'malloc'  was not from MKL , but from system library , like libc

yhu5@kbl01-ub:~/GEMV$ nm -Do /lib/x86_64-linux-gnu/libc.so.6 | grep malloc
/lib/x86_64-linux-gnu/libc.so.6:0000000000086990 T __libc_malloc
/lib/x86_64-linux-gnu/libc.so.6:0000000000086990 T malloc
/lib/x86_64-linux-gnu/libc.so.6:0000000000086b20 W malloc_get_state
/lib/x86_64-linux-gnu/libc.so.6:00000000003c1af0 V __malloc_hook
/lib/x86_64-linux-gnu/libc.so.6:0000000000089c50 W malloc_info
/lib/x86_64-linux-gnu/libc.so.6:00000000003c3790 V __malloc_initialize_hook
/lib/x86_64-linux-gnu/libc.so.6:00000000000868a0 W malloc_set_state
/lib/x86_64-linux-gnu/libc.so.6:00000000000892f0 W malloc_stats
/lib/x86_64-linux-gnu/libc.so.6:0000000000088f00 W malloc_trim
/lib/x86_64-linux-gnu/libc.so.6:0000000000087950 W malloc_usable_size
 

So could you please add such library explicitly in link line , for example

gcc -L$MKL_LIB -fuse-ld=gold t.o -lmkl_rt -ldl -lc -lpthread   etc.

and see if there is any change.

Best Regards,
Ying

[Andrew_T_]

The libc standard library is linked in by default by the linker, but I tried adding it explicitly anyway (-lc), and it had no effect - we still get the same error.

I don't think the link errors are related to pthread, but I also tried including -lpthread (and -pthread) and it had no effect - we still get the same error.

To be clear, all the other third party libraries and our own libraries link fine.  It is only MKL that has this issue.  I suspect libmkl_rt.so has some non-standard property in its symbol table that is causing this issue, and it is exposed by using gold/CentOS/devtoolset-6.

So that you can reproduce the problem locally, I have created a small docker image that demonstrates the issue.  Find attached mkldocker.tar.gz

To use it:

     $ tar xvf mkldocker.tar.gz

     $ cd mkldocker

     $ docker build -t mkldocker .

     $ docker run -ti mkldocker

     # bash build.sh

    ./libmkl_rt.so: error: undefined reference to 'calloc'
    ./libmkl_rt.so: error: undefined reference to 'realloc'
    ./libmkl_rt.so: error: undefined reference to 'malloc'
    ./libmkl_rt.so: error: undefined reference to 'free'

[Andrew_T_]

I have also posted the docker here:

     $ wget http://www.tomazos.com/mkldocker.tar.gz

[Ying_H_Intel]

Hi Andrew,

I haven't centos system so can't check.  But could you please  check if the symbol "malloc" was in your libc library?

or when gold linker, check with libc library it will use?

nm -Do /lib/x86_64-linux-gnu/libc.so.6 | grep malloc

Best Regards,

Ying

[Andrew_T_]

Hi Ying,

The step-by-step instructions I gave to reproduce the issue in the previous message will work from any linux system, you do not need a centos system.  The reason is that docker will transparently download and run a centos system in a virtualized sandbox for you.  After you follow the instructions you will be left on a bash prompt inside that virtualized sandbox.  You can then run whatever commands you see fit to debug the Intel MKL fault (including, for example, running nm on libc.so to see if it provides malloc).

Regards,

Andrew.

[Ying_H_Intel]

Hi Andrew,

Maybe becasue my network issue, I have problem to make the docker to run correctly. We did test several other machines (including Ubuntu, Redhad etc)  at first, we can't reproduce the problem.  But i find way to reproduce your issue. The key is the toolchain.  here is what i find:

On one  machine running Rad hat,

case 1:  originally I had gcc   gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC).  the default ld support gold linker already

 The command  gcc -L. -fuse-ld=gold t.o -lmkl_rt -ldl  just run fine.

yhu5@hsw-ep01 mkldocker]$ ldd a.out
        linux-vdso.so.1 =>  (0x00007ffdd47eb000)
        libmkl_rt.so => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa2b02b6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa2afef5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa2b04e5000)

Case 2:  

 yum install -y devtoolset-6-toolchain

. /opt/rh/devtoolset-6/enable

[root@hsw-ep01 mkldocker]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.2.1/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-6/root/usr --mandir=/opt/rh/devtoolset-6/root/usr/share/man --infodir=/opt/rh/devtoolset-6/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --disable-libgcj --with-default-libstdcxx-abi=gcc4-compatible --with-isl=/builddir/build/BUILD/gcc-6.2.1-20160916/obj-x86_64-redhat-linux/isl-install --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 6.2.1 20160916 (Red Hat 6.2.1-3) (GCC)

[root@hsw-ep01 mkldocker]# gcc -L. -fuse-ld=gold t.c -lmkl_rt -ldl
./libmkl_rt.so: error: undefined reference to 'calloc'
./libmkl_rt.so: error: undefined reference to 'realloc'
./libmkl_rt.so: error: undefined reference to 'malloc'
./libmkl_rt.so: error: undefined reference to 'free'
collect2: error: ld returned 1 exit status
[root@hsw-ep01 mkldocker]# gcc -L. -fuse-ld=gold t.c -lmkl_rt -ldl -lc
./libmkl_rt.so: error: undefined reference to 'calloc'
./libmkl_rt.so: error: undefined reference to 'free'
collect2: error: ld returned 1 exit status

As you see the libc seem solve the symbols 'T' option, but haven't solve the symbol defined with 'W'

[yhu5@hsw-ep01 mkldocker]$ nm -Do /lib64/libc.so.6 | grep malloc                /lib64/libc.so.6:000000000007ff70 T __libc_malloc
/lib64/libc.so.6:000000000007ff70 T malloc
/lib64/libc.so.6:0000000000080170 W malloc_get_state
/lib64/libc.so.6:00000000003ba740 V __malloc_hook
/lib64/libc.so.6:0000000000082580 T malloc_info
/lib64/libc.so.6:00000000003bca40 V __malloc_initialize_hook
/lib64/libc.so.6:0000000000081790 W malloc_set_state
/lib64/libc.so.6:0000000000082340 W malloc_stats
/lib64/libc.so.6:0000000000081fc0 W malloc_trim
/lib64/libc.so.6:0000000000080c10 W malloc_usable_size
[yhu5@hsw-ep01 mkldocker]$ nm -Do /lib64/libc.so.6 | grep calloc
/lib64/libc.so.6:0000000000080960 W calloc
/lib64/libc.so.6:0000000000080960 T __libc_calloc

further, use the workaround:

[root@hsw-ep01 mkldocker]# gcc -L. -fuse-ld=gold t.c -lmkl_rt -ldl -lc -Wl,--allow-shlib-undefined
[root@hsw-ep01 mkldocker]# ldd a.out
        linux-vdso.so.1 =>  (0x00007ffe73184000)
        libmkl_rt.so => /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin/libmkl_rt.so (0x00007fa6a09f6000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa6a07c9000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa6a0408000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa6a1008000)
 

So the issue seems not mkl bug, but  Some specific in higher version of GCC and Gold linker.  I will look the link in details and keep you update if any result.  

Would you please try the command:

gcc -L. -fuse-ld=gold t.c -lmkl_rt -ldl -lc -Wl,--allow-shlib-undefined.  (my test show it should work without problem)

Best Regards,

Ying

[Andrew_T_]

It's great that you have reproduced the issue.

The workaround you have suggested (--allow-shlib-undefined) is unacceptable as it would mean ignoring real correct undefined symbol errors between other libraries at link time, which would show up as run-time errors.

The problem is specific to MKL, because, as I said, we haven't had any problems with any other third party libraries.  It looks like because of this bug it only occurs with devtoolset-6+gold in combination with MKL.  My guess is that libmkl_rt.so is compiled with icc, and icc has produced a symbol table with something unusual in it, and that unusual thing is incompatible with devtoolset-6+gold.  As devtoolset-6 is becoming the standard toolchain on the whole Red Hat family of Linux distributions, and gold is becoming prefered as the standard linker (as it is much faster than the venerable ld.bfd), I think it is essential to make MKL compatible with it.

So we would expect Intel to keep investigating to identify exactly what is unusual about the libmkl_rt.so symbol table that is causing this error.

[Ying_H_Intel]

Hi Andrew,

yes, we will try to find the root cause and update you if any results.

Best Regards,

Ying

[Ying_H_Intel]

Hi Andrew,

We may found the cause.  The issue seem no relate to MKL internal, this is how gold linker behaves.

We saw  the gold linker bug (Gold linker does not resolve symbols using indirect dependencies) is closed as WONTFIX.

Here is some details:

When i try to solve the unresolved symbols (malloc, calloc, realloc free)  by add -lc , which is supposed to work, but still have calloc and free unresolved.  Thing is, when we’re linking like below:

                $ gcc -L$MKL_LIB -fuse-ld=gold t.o -lmkl_rt -ldl -lc

gold linker, while resolving symbols, only takes the first-in-chain library that contains it, in this case it’s libdl – but it’s also U:

$ nm -gA /lib64/libdl-2.17.so|grep -E " (malloc|realloc|calloc|free)"

/lib64/libdl-2.17.so:                 U calloc@@GLIBC_2.2.5

/lib64/libdl-2.17.so:                 U free@@GLIBC_2.2.5

That’s why we saw “undefined reference” for calloc and free after added -ldl and -lc and didn’t see it for malloc and and realloc which were resolved in libc.

This behavior is opposite to regular ld linker which goes thru all libs.

Also for gold linker you need to supply all libs, which symbols were used directly; it doesn’t find it “automagically”.

Symbol	libmkl_rt.so	libdl.so	libc.so
calloc	U	U	W
realloc	U		T
malloc	U		T
Free	U	U	T

So to solve  problem, just add -lc  direct behind of -lmkl_rt  to linker like (switch –ldl and –lc options to linker)

                $ gcc -L. -fuse-ld=gold t.o -lmkl_rt -lc -ldl

​Please let us know if it work for you.

​Best Regard

Ying