offload error: cannot find offload entry?

[wan_b_]

hi

 when I try to run  a fortran progran(compiled by mpiifort with options -offload-build -openmp) on MIC,

it appears two error :

offload error: cannot find offload entry __offload_entry_aoint_2e_dscf_F90_183aoint_2e_dscf_

offload error: process on the device 1 unexpectedly exited with code 1

I  sure the program works well with openmp ,

and I test some MIC examples on my machine to make sure the MIC works.

now I cannot find the problem, can any one help me?

thank a lot.

[Sumedh_N_Intel]

Hi,

From your question, it is unclear if your OpenMP code that worked was a host only or a host+offload code.

From what you have described, I suspect your app's build procedure creates or uses static archives and one must contain your routine with the offload section; however, if the archive containing that routine was not built using xiar -qoffload-build then the required MIC specific archive is not created. While your app links, the MIC-image is missing the MIC-instance of the routine, so at execution time the host-side app reaches the offload, loads the MIC-image, but the run-time cannot find the routine because it does not exist.

You can visit this compiler reference on how to create offload libraires using xiar and xild. 

Thanks, 

Sumedh

[wan_b_]

hi， sumedh

my code is a host-offload code. I want to compute a loop on the MIC with openmp, the openmp code works well on the CPU, but when I add the instruction"!DIR$ offload target(mic) in(......)"  (I have simplify the instruction to prevent other problems), it turns out the error : cannot find offload entry.

And I recompile the linking library like Global Array with mpiifort,  but it didn't work.

I will try xiar and xild to create offload libraries and check whether it works.

thanks a lot.

[wan_b_]

hi  sumedh

thanks for your reply.

I am sorry that I have replied to you earlier but I submited to the furoms failed twice

 and all the feedbacks I've write to you were gone..

Now I follow your suggested, and an example I writed can run on the MIC.

 With the xiar, it created the MIC library, but my code still cannot work on the MIC,

 it turns out that "On the remote process, dlopen() failed. The error message sent back from the sink is /tmp/coi_procs/1/5598/load_lib/ifortoutUqcXvl: undefined symbol: exchange_ On the sink, dlopen() returned NULL. The result of dlerror() is "/tmp/coi_procs/1/5598/load_lib/ifortoutUqcXvl: undefined symbol: exchange_"

offload error: cannot load library to the device 0 (error code 20)"

I think that there are still some other libraries that my code needed ,

and they are not compiled nor on the MIC, I will try to find them out.

I have another question :can I call subroutine in an offload section, does the subroutine affect the MIC to run the program?

I am confused about this.

whatever, I have made some progress, I will update the problems.

thanks very much

wan bo

[Sumedh_N_Intel]

Hi, 

I am not sure what can cause that particular error. Let me look around a little more and I'll get back to you with what I find. 

In the mean time, I have some answers. Yes, you can call a subroutine from an offloaded section. However, for a subroutine to be callable from an offload region, it should be marked with attributes offload. This lets the compiler know that this function is going to be called from an offload and hence the function is compiled for both the host as well as the coprocessor. You can find more about attributes offload at this compiler reference. 

-Sumedh

[Kevin_D_Intel]

First, please drop usage of -offload-build  That is not a valid compiler option.

If the offload directive added for the loop is not in code resident in a static library then using xiar/xild w/-qoffload-build is not necessary.

The undefined symbol exchange_ indicates there no instance of the routine within the offload image. That suggests the routine was not declared for offload via a !DIR$ ATTRIBUTES OFFLOAD or equivalent. Calling a subprogram in an offload section requires declaring the subprogram for offload within the calling scope and at the point of definition using the !DIR$ ATTRIBUTES OFFLOAD.

If you have not already browsed the Fortran MIC samples (/opt/intel/composer_xe_2013/Samples/en_US/Fortran/mic_samples), there are samples demonstrating openmp with offload and calling subprograms within offload code that may help with your interests.

[wan_b_]

Kevin Davis (Intel) wrote:

First, please drop usage of -offload-build  That is not a valid compiler option.

If the offload directive added for the loop is not in code resident in a static library then using xiar/xild w/-qoffload-build is not necessary.

The undefined symbol exchange_ indicates there no instance of the routine within the offload image. That suggests the routine was not declared for offload via a !DIR$ ATTRIBUTES OFFLOAD or equivalent. Calling a subprogram in an offload section requires declaring the subprogram for offload within the calling scope and at the point of definition using the !DIR$ ATTRIBUTES OFFLOAD.

If you have not already browsed the Fortran MIC samples (/opt/intel/composer_xe_2013/Samples/en_US/Fortran/mic_samples), there are samples demonstrating openmp with offload and calling subprograms within offload code that may help with your interests.

Hi kevin, I have declared the subprogramwithin the calling scope and at the point of the definition using the  !DIR$ ATTRIBUTES OFFLOAD.

Then I  declare the subprograms that I used in offload code just as follows 

interface
 !DIR$ OPTIONS /offload_attribute_target=mic
 subroutine leoF09_subprograms_sub(ival)
 integer, intent(in) :: ival
 end subroutine leoF09_subprograms_sub
 !DIR$ END OPTIONS
end interface

But when I run the code on the MIC,  it turns out the same error massage as before:

On the remote process, dlopen() failed. The error message sent back from the sink is /tmp/coi_procs/1/36074/load_lib/ifortout9qYrzO: undefined symbol: exchange_
On the sink, dlopen() returned NULL. The result of dlerror() is "/tmp/coi_procs/1/36074/load_lib/ifortout9qYrzO: undefined symbol: exchange_"
offload error: cannot load library to the device 0 (error code 20)

I can't find the symbol exchange_, I just use a subroutine exchange() to change two variables' position. 

I still don't know the library in "cannot load library to the device 0"   indicate which library.

[Sumedh_N_Intel]

Hi Wan, 

Each routine that you call within an offload should be declared for offload using !DIR$ ATTRIBUTES OFFLOAD. My guess is that you are calling the exchange routine in the offloaded section of the code but the exchange routine at its definition has not been declared for an offload. If you declare the exchange routine for offload, I think the error should go away. 

-Sumedh

[Kevin_D_Intel]

I agree w/Sumedh. The "library" in the error is simply referring to the mic executable which is a shared library. All the symbols referenced in the mic executable (library) must be resolved when loading to the card. The case here is exchange_ is not resolved.

[robert-reed]

Hi Wan.  In a private message to me you asked about the techniques I published in another forum thread to diagnose mute failures to execute the offload code segment and indicated that you'd tried the published methods but they did not work.  You did not explain what happened.  I would recommend that you try again adding the following string to your compiles:  -offload-option,mic,compiler,"-z defs"

With that as an option in the compile, you should get all the undefined symbols remaining in your program before the offload occurs.  Then you might see any functions that have not been attributed as an offload function and follow the advice Sumedh and Kevin have given above to ensure that all the procedures and functions needed for the offload segment are generated for the offload segment.

[Ravi_N_Intel]

use the following option to get the detect the unresolved symbols during link time

-offload-option,mic,ld,"-z defs"

[Kevin_D_Intel]

With Fortran it is a bit easier to determine because names are not as mangled.

Wan, you should find your routine exchange requires decoration for offload as Sumedh indicated.

Robert’s form requires the option to be: “-Wl,-zdefs” (since it goes through the compiler driver). Ravi’s form directs the option to the linker (and not the compiler). Just wanting it to be clear.

[wan_b_]

thanks so much for the help from  Sumedh, Kevin, Robert-reed and Ravi, I'm too sorry that I give the feedbackso late. The MIC platform I use upgraded hardware these days.

I declare the routines, but it still comes out that "undefined symbol: exchange_", but I have rewrite the subroutine so there is no toutine of exchange. This happened every time before I delete the libpscfMIC.a. It's the offload library when I followed Sumedh's advice and used the xiar to compile and it creates this. After I deleted it, the unknown exchange_ disappeard. 

So when I  link without xiar(just  scr), it displays

aoint_2e_dscf.F90:(.text+0x64d): undefined reference to `__offload_target_acquire'

aoint_2e_dscf.F90:(.text+0x887): undefined reference to `__offload_offload'

when I link with xiar, I displays many undefined reference error that I need to add attributes offload

so I want to know whether I need to create the offload library?  And I find little information about  `__offload_target_acquire'  and `__offload_offload'

 wan bo

[Dana_H_]

I am having the same error. So I used -offload-option,mic,ld,'-Wl,-zdefs'.  I see the my first undefined refernce .../fun3d-12.2-63280/Linux-mpi-mic/main.F90:1196: undefined reference to `multigrid_defs_mp_fmg_levels_request_'

I trace it down to the directory and library. I see that I built the object files with ifort (no offload in for library) and used xiar

I see that when ever I use xiar, it indicated that it is using 'ar'.

/usr/local/intel/Composer/composer_xe_2013.1.117/bin/intel64/xiar cru a1.o a2.o ... an.o

xiar: executing 'ar'

Or when I do a which xiar ...

../FUN3D_90/$which xiar
/usr/local/intel/Composer/composer_xe_2013.1.117/bin/intel64/xiar

This is expected, or is there another location for xiar for this purpose.

-dana-

[Dana_H_]

To clarify, when I use xiar (or even xiar -V), the resulting output indicates "xiar: executing 'ar'".

../LibF90/$/usr/local/intel/Composer/composer_xe_2013.1.117/bin/intel64/xiar -V
xiar: executing 'ar'
GNU ar (GNU Binutils; SUSE Linux Enterprise 11) 2.20.0.20100122-0.7.9
Copyright 2009 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

So is xiar working as expected?  Can I use a ranlib on a .a resulting from xiar?

-dana-

[Dana_H_]

I see now that I have to use the -qoffload-build on th xiar command

xiar -qoffload-build cru libCapDefs.a

then I see two

xiar: executing 'ar'
xiar: executing 'ar'

and I get two libraries

-rw------- 1 dphammon g0761  362082 2013-04-01 11:43 libCapDefs.a
-rw------- 1 dphammon g0761       8 2013-04-01 11:43 libCapDefsMIC.a

Let me try throughout my libraries.

[Ravi_N_Intel]

Another quick way to mark all the routines and data in your code with offload_attribute is using the command line option

-offload-attribute-target=mic

This is usefule especially if you want to build libraries code to support both host and MIC

[Shane_R_]

Any further information on the missing "__offload_target_acquire"  and "__offload_offload"?

[Kevin_D_Intel]

I'm not sure what information you are asking about. Could you elaborate?

The missing routines mentioned in earlier replies suggest the offload run-time library was somehow not included in the link.

[baohui_f_]

hello,

my program has the similar error.The error message is:

on the remote process,dlopen() failed. The error sent back from the sink is /tmp/coi_process/1/162729/load_lib/ifortoutnLEt27:Undefined symbol:pmgrid_mp_masterproc_ .

offload error:cannot load library to device 0(error code 20)

The difference from wan.b's  error is that pmgrid_mp_masterproc_ is not a function name or a subroutine name.It is made up of a module name pmgrid and a variable masterproc declared in module pmgrid.what's more ,I also don't know what the character "mp" in pmgrid_mp_masterproc_  is.

The following is my added offload section:

!dec$ offload target(mic) in(arr)

!dec$  omp  parallel do

 do j=1,5000

      do i=5000

       arr(i)=i*3.0/x

     end do

end  do

!$omp end parallel do

[Kevin_D_Intel]

"mp" is part of the compiler provided name decoration associated a module procedures/functions/variables.

The error suggests a reference of masterproc inside offloaded code where the variable lacks declaration for such sue although the code snippet shown doesn't have evidence of that so its hard to explain with only seeing the snippet. Were there no other edits made related to offloading?

In module pmgrid, near the declaration for masterproc, try adding:

!DIR$ ATTRIBUTES OFFLOAD : mic :: masterproc