Problem linking when offloading STL

[John_F_1]

We have some offload code that doesn't run because dlopen() fails with undefined symbols. One of the suggestions that I found was to add "-Wl,-zdefs" to the offload linker options, which might bring the undefined symbols to our attention at link time.  However, when I do this, even the most simple offload code won't link.

For example:

[cpp]

#include <stdio.h>

#pragma offload_attribute(push target(mic))
#include <vector>

struct Foo
{
    std::vector<int> bar;
};
#pragma offload_attribute(pop)

int main(int argc, char * argv[])
{
#pragma offload target(mic:0)
    {
        printf("Hello world");
        Foo* s = new Foo();
    }
}

[/cpp]

I get the following output:

1>------ Build started: Project: TestOffloadLinkage (Intel C++ 14.0), Configuration: Release x64 ------
1>Build started 3/18/2014 10:54:55 AM.
1>InitializeBuildStatus:
1>  Touching "x64\Release\TestOffloadLinkage.unsuccessfulbuild".
1>MessageBuildingWithCompiler:
1>  Building with Intel(R) C++ Compiler XE 14.0
1>ClCompile:
1>  ***** ClCompile (x64 - Intel C++)
1>  Main.cpp
1>Link:
1>  C:\Users\Me\AppData\Local\Temp\ipo_7292.o: In function `std::_Bit_iterator_base::_M_bump_up()':
1>  C:\Users\Me\Documents\Sources\TestOffloadLinkage/Main.cpp:17: undefined reference to `__kmpc_global_thread_num'
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Platforms\x64\PlatformToolsets\Intel C++ Compiler XE 14.0\Microsoft.Cpp.x64.Intel C++ Compiler XE 14.0.targets(977,5): error MSB6006: "xilink.exe" exited with code 1.
1>
1>Build FAILED.
1>
1>Time Elapsed 00:00:01.87

This is really puzzling to me, as I'm not using any threads in this code. I do have /Qopenmp set, and that does seem to be important as it links fine without it. Is this a bug?

[TimP]

I've been unsuccessful in finding out what is the distinction between "-Wl,-zdefs" and -offload-option,mic,compiler,"-z defs" although I could guess that the former makes more sense when building with -mmic.

Setting /Qopenmp causes your code to initialize OpenMP structures even though you don't have any OpenMP directives, so the dependence on openmp library is expected, but not the failure to include it in the link step (unless you didn't use it consistently).  I haven't attempted Windows hosted MIC, but usually the option /# is used to get a verbose echo including steps used in link.

[Kevin_D_Intel]

This is a defect with xilink. I reproduced this when building under the MSVS IDE only and I have not been able to find a work around under the IDE. I reported this to Development (see internal tracking id below) and will keep the post updated with status of a fix.

If there's any possibility that you can link outside the IDE using the icl driver then you can add the /Qoffload-option,mic,link,"-zdefs" option to help identify those unresolved externals you noted occur at run-time.

> icl /Qopenmp /GR- *.obj /Qoffload-option,mic,link,"-zdefs"
Intel(R) C++ Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 14.0.2.176 Build 20140130
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

bar2MIC.o: In function `bar1()':
bar2.cpp:(.text+0x6): undefined reference to `bar2()'
icl: error #10104: unable to open 'bar2MIC.exe'

(Internal tracking id: DPD200254589)

[Kevin_D_Intel]

Actually, toying with a bit more, I was wrong earlier. This is appears to be working as designed and the xilink failure is expected because the -zdefs disallows unresolved externals to exist in the MIC-side executable; therefore, the MIC-side link is failing which is what xilink reports.

The unresolved references are actually identified before the xilink failure. For the Release build, disable IPO and (Configuration Properties > C/C++ / Optimization [Intel C++] > Interprocedural Optimization) and it becomes obvious the unresolved references are found in the <file>MIC.o object file as noted below.

1>------ Rebuild All started: Project: ConsoleApplication2, Configuration: Release x64 ------
1>  t1.cpp
1>  x64\Release\t1MIC.o: In function `main':
1>  C:\Temp\kddavis\ConsoleApplication2/t1.cpp:17: undefined reference to `__kmpc_global_thread_num'
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Platforms\x64\PlatformToolsets\Intel C++ Compiler XE 14.0\Toolset.targets(997,5): error MSB6006: "xilink.exe" exited with code 1.
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

For the Additional Options for MIC Offload Linker, only specify: -zdefs

Something else to try is, instead of compiling with /Qopenmp use /Qopenmp_stubs  (Configuration Properties > C/C++ / Language [Intel C++] > OpenMP Support) just to help identify the unresolved reference and avoid the unresolved reference to the __kmpc_global_thread_num.

Let me know whether this helps.

[John_F_1]

Hi Kevin,

The reason I started using this option is because everything would compile fine, but when I tried running the offloaded code I would get errors like:

On the remote process, dlopen() failed. The error message sent back from the sink is /var/volatile/tmp/coi_procs/1/6745/load_lib/driverMIC.out: undefined symbol: ...

So, I am looking for a nice way to detect (and correct) this through the IDE rather than at runtime.  What I'm trying to understand is why the MIC-side link is failing with -zdefs. Removing this option, it compiles/links and runs properly, so there evidently isn't anything wrong with the build but perhaps rather a difference between the environment at build time vs. runtime.

Perhaps you can suggest a better way to detect (truly) undefined symbols in the MIC offload regions?  Thanks for your help,

[John_F_1]

And maybe it isn't 100% clear from my comments: the sample code I provided runs fine.  I encounter the dlopen() failure periodically during the development process, and would like to catch it during the build process.

[Kevin_D_Intel]

I'm sorry if I created more confusion with my earlier replies.

The -zdefs option should provide what you are looking for. All unresolved references in the MIC-side image will be reported at link time and the link will fail if any exist.

When using the -zdefs (I also verified the -Wl,-zdefs form works) under the IDE in the Additional Options for MIC Offload Linker setting, xilink invokes the associated Linux linker (ld) to perform the link for the MIC-side. By design, ld's -zdefs option disallows the resulting MIC-side image from being created with unresolved references. So, it reports all unresolved references and returns an error code to xilink which in turn reports the failure under the IDE. This is the expected behavior when unresolved references exist. If none exist, the link will succeed.

I'm still trying to understand why with /Openmp the OpenMP routine is not resolved in your example but I believe you should be able to use -zdefs despite that routine being unresolved. What I was trying to say in my last reply was that if that OpenMP routine is hindering using -zdefs then if you use /Qopenmp_stubs, all Openmp references should be resolved in the MIC-side image and any unresolved references in within only your code should only be reported.

[John_F_1]

Thanks for the explanation, it is much clearer now.

[jimdempseyatthecove]

Kevin,

I have a related problem that I posted elsewhere.

C# console application calling a C++ dll containing an offload region, containing openmp region. This works.

To this project I add my DLL (Fortran) and I add to the C++ dll:

extern "C" int crames_get_num_mics();

extern "C" int crames_try_offload_mic(int iMIC);

and

std::cout << "crames_get_num_mics() = " << crames_get_num_mics() << std::endl;

fflush(0);

std::cout << "crames_try_offload_mic(0) = " << crames_try_offload_mic(0) << std::endl;

fflush(0);

There are several other files inside the library. When the library is built as a static library, all the functions work from a test console Fortran application. I need the Fortran library to be a DLL and be callable from C# to the C++ DLL then to my Fortran DLL.

The get_num_mics calls OFFLOAD_NUMBER_OF_DEVICES

and the try_offload_mic

INTEGER FUNCTION internal_crames_try_offload_mic(iMIC)
    use mic_lib
    INTEGER :: iMIC
    !!DEC$ ATTRIBUTES VALUE:: iMIC
    integer :: resultOut
    resultOut = 9999
    !dir$ offload begin target (mic:iMIC) in(iMIC) out(result)
    !dir$ if defined(__MIC__)
    resultOut = 0 ! Offload_get_device_number()
    !dir$ else
    resultOut = -1
    !dir$ endif
    !dir$ end offload
    internal_crames_try_offload_mic = resultOut
    return
    END FUNCTION internal_crames_try_offload_mic

When I insert that code above I get:

_Offload_number_of_devices() = 1
crames_get_num_mics() = 1
crames_try_offload_mic(0) = On the remote process, dlopen() failed. The error me
ssage sent back from the sink is /var/volatile/tmp/coi_procs/1/7949/load_lib/930
82315MIC.out: undefined symbol: for_check_mult_overflow64
On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/t
mp/coi_procs/1/7949/load_lib/93082315MIC.out: undefined symbol: for_check_mult_o
verflow64"
offload error: cannot load library to the device 0 (error code 20)
Press any key to continue . . .

The routine for_check_mult_overflow64 is one of your runtime library routines. The offload dynamic library is not loading this, nor did the code come in as part of the "fat" DLL routines.

When I remove my other (non-called) subroutines and functions, the two calls work

This has had me stuck for several days.

Jim Dempsey

[jimdempseyatthecove]

In performing dumpbin on the windows .dll's I find that for_check_mult_overflow64 is located in libifcoremdd.dll

Therefore, by inference, one might assume for_check_mult_overflow64 to be located in either:

libifcoremt_pic.a, libifcoremt.so libifcoremt.so.5

How do I request that the .so library, that will be called from my offload, will also be ported into the MIC when the offload is first loaded?

Note, when my .DLL is called from a Fortran PROGRAM, the .so library is apparently copied, since this works.

When my .DLL is called from a C++ or C# program, then when the first offload occurs, I get the error.

Ergo, the .so library was not transported upon first entry into the offload.

Jim