Offload static library problem (error 20)

[Colm_C_]

Hi,

I'm having trouble running some offload code. The code compiled fine and appeared to execute until it needed some functions that were defined as MIC only and threw the following error:

[Offload] [HOST]  [State]   Initialize logical card 0 = physical card 0
On the remote process, dlopen() failed. The error message sent back from the sink is /var/volatile/tmp/coi_procs/1/7016/load_lib/iccoutyjfaoE: undefined symbol: bfgsfit_visibilities_mic
offload error: cannot load library to the device 0 (error code 20)
On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/tmp/coi_procs/1/7016/load_lib/iccoutyjfaoE: undefined symbol: bfgsfit_visibilities_mic"
[Offload] [HOST]  [State]   Unregister data tables

I read about some similar issues on the forum and it seems to me that there is some kind of linking problem. A library made with xiar (and containing the MIC code) should be linked by the following makefile:

OUTPUT=
CXX=icpc

$MKLROOT=/opt/intel/composer_xe_2013.5.192/mkl
IFACE_LIB=mkl_intel_lp64
THREADING_LIB=mkl_intel_thread
CORE_LIB=mkl_core

LDFLAGS=-L$(MKLROOT)/lib/intel64 -l$(IFACE_LIB) -l$(THREADING_LIB) -l$(CORE_LIB) -lpthread
MIC_LDFLAGS=-L$(MKLROOT)/lib/mic -l$(IFACE_LIB) -l$(THREADING_LIB) -l$(CORE_LIB) -L$(MIC_LD_LIBRARY_PATH)

#MICFLAGS =-offload-option,mic,compiler,"-DUSE_MIC -vec-report1 -g -O0 -Wall"
MICFLAGS =-offload-option,mic,compiler,"-DUSE_MIC -vec-report1 -O3 -openmp -Wall"
#MICFLAGS =-offload-option,mic,compiler,"-DUSE_MIC -O1 -profile-functions -profile-loops=all -profile-loops-report=2 -openmp"
MICLDFLAGS=-offload-option,mic,ld,"$(MIC_LDFLAGS)"

CXXFLAGS +=-DUSE_MIC -g -O3 -Wall -openmp -vec-report=1 $(MICFLAGS)
#CXXFLAGS +=-DUSE_MIC  -O1 -profile-functions -profile-loops=all -profile-loops-report=2 -openmp -vec-report=1 $(MICFLAGS)

CASA_LIBDIR=/home/coughlan/phi_stuff/lofar_software_install/lib
CASA_INCDIR=/home/coughlan/phi_stuff/lofar_software_install/include/casacore
CASA_LIBS=-lcasa_casa -lcasa_tables -lcasa_measures -lcasa_ms -lcfitsio

#LDFLAGS=-Wl,-t,--rpath,/software/users/lofareor/SW/lib64
# -Wl,--hash-style=both

# with multithread FFTW
MY_LIBS=-lsagecal
INCLUDES=-I. -I./lib -I$(CASA_INCDIR) -I/usr/include
LIBPATH=-L$(CASA_LIBDIR) -L./lib

#### glib
GLIBI=-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/lib64/glib-2.0/include/
GLIBL=-lglib-2.0

OBJECTS=main.o data.o
default:sagecal
main.o:main.cpp
    $(CXX) $(CXXFLAGS) $(INCLUDES) $(GLIBI) -c $<
data.o:data.cpp data.h
    $(CXX) $(CXXFLAGS) $(INCLUDES) $(GLIBI) -c $<
sagecal:$(OBJECTS) ./lib/libsagecal.a
    $(CXX) $(CXXFLAGS) $(LDFLAGS) $(MICFLAGS) $(MICLDFLAGS) $(INCLUDES) $(GLIBI) $(LIBPATH)  -o $@  $(OBJECTS) $(MY_LIBS) $(CASA_LIBS)  $(GLIBL)
clean:
    rm *.o *.tmp *.fits

When I look at the library with nm I get

bash-4.1$ nm -C ./lib/libsagecal.a| grep vis
0000000000002080 T predict_visibilities
0000000000003fc0 T bfgsfit_visibilities
0000000000005e30 T bfgsfit_visibilities_mic
00000000000026c0 T sagefit_visibilities
00000000000047a0 T sagefit_visibilities_mic
0000000000005220 T predict_visibilities_multifreq
0000000000004a50 t visibilities_threadfn_multifreq
bash-4.1$ nm -C sagecal| grep vis
000000000043b1e0 T predict_visibilities
0000000000443110 T predict_visibilities_multifreq
0000000000442940 t visibilities_threadfn_multifreq

The function mentioned in the error (bfgsfit_visibilities_mic) is missing from the final executable. Is this the source of the problem (or is the compiled code somehow located elsewhere like the MIC.o files). If not, could you give me some pointers as to what might be the issue?

Thanks for your help,

Colm

[Kevin_D_Intel]

The static library containing routines with offload must be built with xiar and the -qoffload-build option.  The makefile details do not show how the static archive is built.

If you have not already see this, the User guide section About Creating Offload Libraries with xiar and xild may help.

[Colm_C_]

Thanks for the quick response. The library was build with those options as far as I can see. here is the makefile for the library - the MIC version of the library is also generated and the libsagecal.a file has the right MIC functions in it according to nm.

CC=icc
CXX=icpc
LD=icc

# MKL
#MKLROOT=/opt/intel/composer_xe_2013.5.192/mkl
IFACE_LIB=mkl_intel_lp64
THREADING_LIB=mkl_intel_thread
CORE_LIB=mkl_core

LDFLAGS=-L$(MKLROOT)/lib/intel64 -l$(IFACE_LIB) -l$(THREADING_LIB) -l$(CORE_LIB) -lpthread -lm
MIC_LDFLAGS=-L$(MKLROOT)/lib/mic -l$(IFACE_LIB) -l$(THREADING_LIB) -l$(CORE_LIB)

##CFLAGS +=-DUSE_MIC -Wall -DDEBUG -g -O0 -openmp -vec-report=1
#CFLAGS +=-DUSE_MIC -Wall -O1 -profile-functions -profile-loops=all -profile-loops-report=2 -openmp
CFLAGS +=-DUSE_MIC -Wall -O3 -openmp -vec-report=1
#MICFLAGS =-offload-option,mic,compiler,"-DUSE_MIC -vec-report1 -g -O0 -Wall"
MICFLAGS =-offload-option,mic,compiler,"-DUSE_MIC -vec-report1 -O3 -openmp -Wall"
#MICFLAGS =-offload-option,mic,compiler,"-DUSE_MIC -O1 -profile-functions -profile-loops=all -profile-loops-report=2 -openmp"
MICLDFLAGS=-offload-option,mic,ld,"$(MIC_LDFLAGS)"

#LAPACK=-L/usr/lib/atlas/sse -llapack -lblas
#LAPACK=-L/usr/local/GotoBLAS2/lib -lgoto2 -lpthread -lgfortran
LAPACK=-L/usr/local/OpenBLAS/lib/ -lopenblas -lgfortran -lpthread

INCLUDES= -I.
LIBPATH=

#### glib
GLIBI=-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/lib64/glib-2.0/include/
GLIBL=-lglib-2.0

OBJECTS=readsky.o dataio.o predict.o lmfit_nocuda.o clmfit_nocuda.o lbfgs_nocuda.o myblas.o residual.o robustlm.o updatenu.o robust_lbfgs_nocuda.o
#  clmfit_nocudaMIC.o lmfit_nocudaMIC.o robust_lbfgs_nocudaMIC.o updatenuMIC.o\
lbfgs_nocudaMIC.o myblasMIC.o robustlmMIC.o

default:libsagecal.a
readsky.o:readsky.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
dataio.o:dataio.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
predict.o:predict.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
lmfit_nocuda.o:lmfit_nocuda.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
clmfit_nocuda.o:clmfit_nocuda.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
lbfgs_nocuda.o:lbfgs_nocuda.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
myblas.o:myblas.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
residual.o:residual.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
robustlm.o:robustlm.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
updatenu.o:updatenu.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<
robust_lbfgs_nocuda.o:robust_lbfgs_nocuda.c
    $(CC) $(CFLAGS) $(INCLUDES) $(GLIBI)  -c $<

RANLIB=ranlib
libsagecal.a:$(OBJECTS) sagecal.h
    xiar -qoffload-build rcs $@ $(OBJECTS); \
    $(RANLIB) $@;

[Colm_C_]

As the library did not seem to be linking properly I just moved all the object and MIC object files into the directory with the source code and changed the makefile to link them directly. Running nm on the executable shows that the missing MIC functions are now present and the code works as expect. I'm still not sure what I was doing wrong (the library was being built with the options mentioned), but at least it works. Thanks for your help - if you've any idea why the library doesn't work I'd still be interested to know.