Can't build shared libraries via libtool

[Allin_C_]

I'm trying to port a project as a native Xeon Phi application. The project depends on some third-party libraries (GLib, libxml2, libgmp), and I understand that I have to cross-compile these libraries for x86_64-k1om-linux. I'm using the article "Autotools and Intel® Xeon Phi Coprocessor" as a guide, but here's the problem I've hit. I need shared libs, but I find that libtool doesn't think that icc in -mmic cross-mode can create them. I'm exporting CC=icc and LD=icc, and placing -mmic in CFLAGS and LDFLAGS as recommended, as well as passing --host=x86_64-k1om-linux to ./configure, but I'm seeing

checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-k1om-linux-gnu
checking for ld used by icc... icc
checking if the linker (icc) is GNU ld... no
...
checking for icc option to produce PIC... -fPIC -DPIC
checking if icc PIC flag -fPIC -DPIC works... yes
...
checking whether the icc linker (icc -m elf_x86_64) supports shared libraries... no

I've tried hacking on the configure script some (e.g. so as to replace the auto-generated "icc -m elf_x86_64" with "icc -mmic -shared") but to no avail. II've also checked, by building a trivial shared library without using libtool, that icc (14.0.1.106 Build 20131008) can in fact do the job. Any help would be much appreciated.

[Kevin_D_Intel]

What does libtool analyze when trying to determine whether the resulting shared library is valid?

Could you perhaps alter that check within the configure?

[Allin_C_]

Kevin Davis (Intel) wrote:

What does libtool analyze when trying to determine whether the resulting shared library is valid?

Could you perhaps alter that check within the configure?

These are good questions, but not so easy to answer! Configure scripts can be quite impenetrable. My current thinking is that the primary source for this part of the script -- namely libtool.m4 -- needs to be modified for current icc in its cross-mode, -mmic.

[Allin_C_]

I'm now convinced the correct solution to this problem requires revising libtool.m4. I'm not competent to do that but in the meantime, in case it's of any use to others, I can offer a piece of horrible hackery that works.

1) Run configure as usual, but including the option --cache-file=k1om.cache

2) Open k1om.cache in an editor, and if the line

  lt_cv_prog_gnu_ld=${lt_cv_prog_gnu_ld=no}

is present, change "no" to "yes".

3) Open the configure script in an editor and look for lines beginning with

  if $LD --help 2>&1 | $EGREP ': supported targets:.* elf'

The first such line occurs within a section pertaining to beos; ignore it and find the second occurrence, and replace it with

  if $LD --help 2>&1 | $GREP 'mmic'

4) Re-run configure and libtool should now be willing to build shared libraries for the mic architecture, when both CC and LD are set to icc.

[Kevin_D_Intel]

Great sleuthing! Thanks for sharing.

[Evan_P_Intel]

If you haven't upgraded to MPSS 3.2 yet, I suggest you do so--if you weren't aware, pre-compiled binary RPMs for libxml2 and libgmp are now provided for Xeon Phi. Glib is more problematic...it typically has a dependency on libffi, a foreign function interface library, which hasn't been ported to Xeon Phi yet. It may be possible to configure out that dependency--I don't know.

It's worth noting that these pre-compiled binaries were compiled by GCC, not ICC, so if your usage of them is performance critical, you may still want to recompile them yourself using ICC. (Recall that the provided GCC port is unsophisticated and produces comparatively low quality object code.)

Compiling using GCC is straightforward; for example:

tar -xf libxml2-2.7.8.tar.gz
cd libxml2-2.7.8
. /opt/mpss/3.2/environment-setup-k1om-mpss-linux
gnu-configize  # to update config.sub
./configure $CONFIGURE_FLAGS --prefix=/usr --libdir=/usr/lib64
make -j 16

The key to this process is sourcing environment-setup-k1om-mpss-linux, which sets a number of environment variables and modifies $PATH. Note also that the value of $CONFIGURE_FLAGS passes "--host=k1om-mpss-linux" to the ./configure script.

Using ICC to compile requires a bit more finesse since it behaves badly when given the "--sysroot" option:

tar -xf libxml2-2.7.8.tar.gz
cd libxml2-2.7.8
. /opt/mpss/3.2/environment-setup-k1om-mpss-linux
gnu-configize  # to update config.sub
./configure $CONFIGURE_FLAGS --prefix=/usr --libdir=/usr/lib64 \
    LDFLAGS='' LD=k1om-mpss-linux-ld CPPFLAGS='' CC=icc CFLAGS='-mmic'
make -j 16

The configure script prints the following while completing successfully (for comparison to the similar quoted output above):

configure: WARNING: unrecognized options: --with-libtool-sysroot
configure: loading site script /opt/mpss/3.2.1/site-config-k1om-mpss-linux
checking build system type... x86_64-pc-linux-gnu
checking host system type... k1om-mpss-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for k1om-mpss-linux-strip... k1om-mpss-linux-strip
...
checking for k1om-mpss-linux-gcc... icc
checking whether the C compiler works... yes
...
checking for ld used by icc... k1om-mpss-linux-ld
...
checking for icc option to produce PIC... -fPIC -DPIC
checking if icc PIC flag -fPIC -DPIC works... yes
...
checking whether the icc linker (k1om-mpss-linux-ld) supports shared libraries... yes

I didn't actually test the ICC-compiled libxml2 I built with this method, but I did verify that the shared objects were created and looked sane. I'd be interested in knowing whether you have any problems.

(EDIT: Added LD=k1om-mpss-linux-ld to ./configure command, though it doesn't make any difference with libxml2. Also added ./configure output.)

[Evan_P_Intel]

To be a little more explicit, the SDK included with MPSS 3.2 contains, among other things, an appropriately modified version of "libtoolize". Upgrading is strongly recommended, as using 3.2 will resolve a lot of issues one might have with cross-compilation of autotoolized opensource software.

[Allin_C_]

Thanks for the suggestion, I will propose an MPSS update to the sysadmin.

I've now properly  tested the following, compiled with icc for mic using the libtool hackery mentioned above:

glib-2.26.1
libxml2-2.9.1
fftw-3.3.4
gmp-6.0.0a

The glib version is old enough that it doesn't require libffi, but recent enough to do all that I need.