undefined symbol in running not in linking of executable linked with a static library (offload model)

Nikos_S_ · ‎09-29-2015

Hello,

I am trying to run an offload application on Xeon Phi, and I get the following error.

"On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/tmp/coi_procs/1/205497/load_lib/iccouthqsEw3: undefined symbol: DISTMEM_rank"
On the remote process, dlopen() failed. The error message sent back from the sink is /var/volatile/tmp/coi_procs/1/205497/load_lib/iccouthqsEw3: undefined symbol: DISTMEM_rank
offload error: cannot load library to the device 0 (error code 20)"

The DISTMEM_rank variable is defined (verified with nm) in a static library that is linked to the executable.

I haven't used the __attribute__((target(mic))) annotation for the variable but I have used the "-offload-attribute-target=mic" option when I built it and so I guess I am fine with that.
Furthermore, I am using the option -offload-option,mic,compiler,"-z defs" to reveal any undefined symbols during the linking phase but everything seems to work fine. I have no output with missing symbols.

From the myldout.id output I get a bunch of symbols defined the same way like the one I am missing, and are said to be "normal symbol". For my missing symbol i get the following:

205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/offload_main [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libcoi_device.so.0 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libimf.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/liboffload.so.5 [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libmyo-service.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libm.so.6 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libiomp5.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libstdc++.so.6 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libgcc_s.so.1 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libpthread.so.0 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libc.so.6 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libdl.so.2 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/librt.so.1 [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libscif.so.0 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/ld-linux-k1om.so.2 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libsvml.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libirng.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libintlc.so.5 [0]
205449: symbol=DISTMEM_rank; lookup in file=/var/volatile/tmp/coi_procs/1/205449/load_lib/iccouthqsEw3 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libimf.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libsvml.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libirng.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libm.so.6 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/liboffload.so.5 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/load_lib/libcilkrts.so.5 [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libstdc++.so.6 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libgcc_s.so.1 [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libintlc.so.5 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libc.so.6 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libdl.so.2 [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libcoi_device.so.0 [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libmyo-service.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/tmp/coi_procs/1/205449/libiomp5.so [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/libpthread.so.0 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/ld-linux-k1om.so.2 [0]
205449: symbol=DISTMEM_rank; lookup in file=/lib64/librt.so.1 [0]
205449: symbol=DISTMEM_rank; lookup in file=/usr/lib64/libscif.so.0 [0]
205449: /var/volatile/tmp/coi_procs/1/205449/load_lib/iccouthqsEw3: error: symbol lookup error: undefined symbol: DISTMEM_rank (fatal)

Can you please explain me what am I missing?

Thank you in advance,

Nikos

I am using the version below:

Kevin_D_Intel · ‎09-29-2015

You very well could be exercising a very old defect in the 13.1 compiler. That is a very old compiler. I would really recommend that you upgrade before spending any more time/effort here. Everything you mentioned doing is exactly the recommended tips/tricks to flesh out what is usually an oversight of not decorating the variable/function accordingly, but it appears you have probably done so in your case, so the fault is probably in the compiler or xiar.

If you haven't already, perhaps you could try adding the explicit __attribute__((target(mic))) to the variable in question.

Just curious, what version of MPSS are you mixing that 13.1 compiler with?

Everything is working fine for linking because the Linux .so is allowed to have undefined references at link-time and thus the link succeeds; however, as you found out, you cannot get past it at run-time.

Nikos_S_ · ‎09-29-2015

Hello Kevin,

Thanks for the quick response. I read about this possible defect but I wanted to ask first before upgrading because I am working in my university's cluster.

I have already tried the explicit annotation of the variable and got the same error.

The mpss version is 3.1.1.

Kevin_D_Intel · ‎09-29-2015

Sorry to hear explicit annotation did not work. You might face other issues down the road by not using a much more current compiler so I would really try to get the compiler upgraded.

I'm not sure at the moment what to suggest you try. Can you alter the declaration of the variable in any way to see if that helps? Make it global or static if not already?

Are you able to share any bit of relevant code? Maybe our Developers can determine what is lacking in 13.1 enough to offer a workaround.

Nikos_S_ · ‎09-29-2015

The variable is global declared in a header file and defined in the linked library, not called directly from my code.

my code is as simple as:

...

void offloadfun( int * d, int k)
{

static const thread_status_t thread_status = SSsingle_threaded;
struct pth_t * const MT_self = (pth_t *)SMTsingleton_queen;
int a;
a = k+40;
*d = a;
return;
}

int main() {

...

#pragma offload target(mic)

offloadfun(&d,k);

...

}

I think that the two variables defined in the offloadfun and are used in the underlying pthread implementation cause the linking with the undefined variable.

Actually, I managed to overcome the error by defining dummy definition of the missing variable inside the file that needs it and do not link with the library in which these variables where defined. Anyway, there were not important for my project. But in the next step I got an error for another missing variable. Is it how it should be that I don't get a single error message with all the undefined symbols but instead I get a new one every time I resolve the previous?

Thank you,

Nikos

Kevin_D_Intel · ‎09-29-2015

At run-time yes, you will hit these one by one but -zdefs should flesh all of them out at link time. Reading back in this post I see you said -zdefs did not flesh any out which I mis-read earlier. Check the syntax that you used. What you posted having used likely did not get passed through.

When passing it through the compiler, the form is: -offload-option,mic,compiler,"-Wl,-zdefs"

When passing directly to the linker only, the form is: -offload-option,mic,ld,"-z defs"