C# calling C++ (managed) DLL with static IVF .lib calling MIC

[jimdempseyatthecove]

I am having an issue making a call to Fortran function with an offload region. First the problem area, then the background:

000007FEE051CF0F  mov         rax,qword ptr [rbp+148h]  
000007FEE051CF16  mov         r8,rax  
    !dir$ offload begin target (mic:iMIC) in(iMIC) out(result)
000007FEE051CF19  mov         r9d,ebx  
000007FEE051CF1C  call        __offload_offload3 (07FEE0634908h)  
000007FEE051CF21  mov         dword ptr [rbp+120h],eax  
    !dir$ if defined(__MIC__)
    resultOut = Offload_get_device_number()
    !dir$ else
    resultOut = -1
    !dir$ endif
    !dir$ end offload
    crames_try_offload_mic = resultOut
000007FEE051CF27  mov         eax,dword ptr [RESULTOUT]  

The function is a trivial function that takes an integer as input, uses it to target a specific MIC (0 in this case). The program crash occurs on the call to __offload_offload3.

If I change the code to use: resultOut = 0 ! Offload_get_device_number()

IOW remove the call to avoid library function issue, the code still crashes when attempting to enter the off load region.

Now for some background information.

The code is called in a circuitous route.

This is a C# application, calling a C# assembly, calling an MS C++ (managed) .DLL (x64), which incorporates the Static Library (x64) built with IVF 2015 update 4.

I've tried compiling the library with and without the MIC Offload Compiler option -fPIC. Crashes either way.

When I build the library without the -fPIC and replace the ( C# application, calling a C# assembly, calling an MS C++ (managed) .DLL (x64)) portion with a dummied up Fortran Console application, the call through the library function works.... with a work around for a quirk.

On a different thread on this forum I posted a message about OFFLOAD_DEVICES being broken.

Prior to the offload region, I make a function call to OFFLOAD_NUMBER_OF_DEVICES. When configuring OFFLOAD_DEVICES=0, the console application successfully makes it through the OFFLOAD_NUMBER_OF_DEVICES without a hang. Removing OFFLOAD_DEVICES or setting it to =0,1 (I have two MIC's), or setting it to =1, causes the console application to take a ~60 second pause, then depending on setting, the subsequent call to the function that enters the offload region crashes.

As an independent test, building the MIC sample program: LEO_tutorial exhibits the same problem. Setting OFFLOAD_DEVICES=0 works without the pause (other than the ~3 second initial code injection). Anything else it hangs for 60 seconds or crashes. Note micsmc shows both devices.

Now back to the C# application, when configured with OFFLOAD_DEVICES=0 (works without hang in console application), I get the 60 second hang before at the call through the C# assembly, calling an MS C++ (managed) .DLL (x64), which incorporates the Static Library (x64) built with IVF 2015 update 4 call chain to the F90 function I provided to obtain the number of mics.

Note, the 60 second hang occurs as if the "OFFLOAD_DEVICES=0" environment variable, which was set, but symptomatically, was as if not present.

Regardless of the pause, I'd of expected to make it through the one line offload that performs resultOut=1

I've tried adding Offload_Report(3)...

but your almost complete documentation on this does not tell you were the report goes, or how you can direct it to a known file. I cannot locate the report file, and the system does not have a printer. I've tried setting MIC_HOST_LOG=C:\Phi\Log but nothing appears in there.

I hope I've given you enough information, such that you can provide me with some hints (at least to be able to generate the offload report to a file).

Jim Dempsey

[jimdempseyatthecove]

I'd like to add that for_rtl_init_ has not been called. I will add that to the code and see what happens....

In addition to adding for_rtl_init_, in the C++ code, I added a GET_ENVIRONMENT_VARIABLE in the Fortran code to check on OFFLOAD_DEVICES. It does read as '0        ' and the hang does not occur. Note, I reset the environment variable OFFLOAD_DEVICES to 0 and restarted VS 2013.

So the hang is eliminated with appropriately set OFFLOAD_DEVICES, but the crash is not.

Jim

[jimdempseyatthecove]

Keeping in mind that this is a DLL in an application with no console, In the working part of the Fortran code (pre-mic) I've opened on units 0 and 6, status='unknown', buffered='no' and call offload_report(3).

my two files are created, but nothing gets written into them, or should I say, nothing appears in either one of them, possibly due to the crash inhibiting the final write (but isn't this what buffered='no' is all about).

Jim Dempsey

[jimdempseyatthecove]

I have a lead into the problem, which I reported on the IFV forum (https://software.intel.com/en-us/forums/topic/559330#comment-1828645) way way down in the messages. The report is more appropriate to this thread.

After a lot of futzing around I got a lead to the issue (related to the above link) but belongs here.

At issue here, are two issues.

a) when building a .DLL (Fortran) for use into a C# forms application make sure there is a way to capture the console output messages. By default they go nowhere. Note, on program abort, in the debugger, the output window immediately reverts back to the build window, if by chance console output to the missing console were displayed there. And now the crux of the matter is...

b) The offload code is making calls to routines in the Intel supplied libraries (e.g. for_check_mult_overflow64) and that the offload variant of these are not loaded in at link time, but are deferred for a later dynamic load, and the dynamic load is unable to resolve the load.

Typical output (with a few diagnostic prints to screen made by me):

_Offload_number_of_devices() = 1
crames_get_num_mics() = 1
crames_try_offload_mic(0) = On the remote process, dlopen() failed. The error me
ssage sent back from the sink is /var/volatile/tmp/coi_procs/1/7837/load_lib/772
02214MIC.out: undefined symbol: for_check_mult_overflow64
On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/t
mp/coi_procs/1/7837/load_lib/77202214MIC.out: undefined symbol: for_check_mult_o
verflow64"
offload error: cannot load library to the device 0 (error code 20)
Press any key to continue . . .

I tried linking in liboffload.lib to no effect. Force Symbol References - no effect. Link Library Dependencies Yes. The additional Library Directories has $(IFORT_COMPILER15)compiler\lib\intel64

Link options:

/OUT:"x64\Debug\meCRAMESparallelDLL.dll" /INCREMENTAL:NO /NOLOGO 
/LIBPATH:"C:\Program Files (x86)\Intel\Composer XE 2015\compiler\lib\intel64" 
/MANIFEST /MANIFESTFILE:"x64\Debug\meCRAMESparallelDLL.dll.intermediate.manifest" 
/MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG
 /PDB:"C:\Phi\ConsoleApplication1_u534852\ConsoleApplication1\meCRAMESparallelDLL\x64\Debug\meCRAMESparallelDLL.pdb"
 /SUBSYSTEM:WINDOWS /IMPLIB:"C:\Phi\ConsoleApplication1_u534852\ConsoleApplication1\meCRAMESparallelDLL\x64\Debug\meCRAMESparallelDLL.lib" 
/DLL /qnoipo

Any hints

[jimdempseyatthecove]

And the Fortran command line options are:

/nologo /debug:full /Od 
/I"C:\Program Files (x86)\Intel\Composer XE 2015\compiler\include\intel64" 
/recursive /reentrancy:threaded
/Qoffload-option,mic,compiler,"-fPIC" 
/module:"x64\Debug\\" /object:"x64\Debug\\"
/Fd"x64\Debug\vc120.pdb" /traceback /check:bounds /check:stack
/libs:dll /threads /dbglibs /c

Jim Dempsey

[P__Robert]

This is very interesting.  Were you able to resolve the issue?

[jimdempseyatthecove]

Not yet,

I got a private post from Frances that she heard from a colleague in the compiler team that there was an issue where when the main program was not IVF (or apparently Intel C++) that all the dependent .dll MIC offload code and/or .so dependency files were not loaded or not findable in the resultant execution image. The colleague said this was fixed, but could not remember when it was fixed, but it should be in the new Beta program.

I downloaded and installed the new 2016 beta and the symptoms changed a little bit. This may be the same underlying problem, a new problem or me just not following hints/instructions hidden away somewhere.

The old symptom was upon first call to offload: Undefined symbol: for_check_mult_overflow64

The new symptom: System.FileLoadException: Unknown Module, A procedure imported by MyCPP.dll could not be loaded.

The above occurs on program load now, as opposed to at first call to the Fortran.DLL containing the offload code.

Unfortunately, in MS's infinite wisdom, they do not provide the module name.

In the MyCPP.dll (not real name), when I comment out the function calls to MyFortran.dll containing to offloads, then the program run (with non-functioning stub routines).

This looks (smells) like a path issue. When I run a simple IVF console program LEO_tutorial from the samples, this works. It boils down to getting the dependencies to get loaded in when building a .DLL that is not called by a IVF PROGRAM or Intel CPP main.

Do you have any suggestions?

Jim Dempsey

[jimdempseyatthecove]

The new symptom was a placement issue of the separate project .dll file. When corrected, the old issue of performing the first offload is still there.

Jim

[jimdempseyatthecove]

The issue is resolved.

I needed to install the V16 (2016 Beta).

But this did not initially resolve the issue. After a long series of go-a-rounds with Intel, (principally Ravi) with his site not being able to reproduce the problem and mine exhibiting the problem it turned out that for my site, I installed the 2016 Beta, followed by the Redistributables. And the redistributables reset the path to the MIC_LD_LIBRARY to that containing %INTEL_REDIST%... which held the location of the Parallel Studio XE 2015 that is installed on my system.

The interesting part (sad part) was this error did not show up with a Console App (Intel C++ or IVF), but did with the C# forms and console applications. This made the problem hard to diagnose.

I wish to thank Ravi and Frances for the time they spent trying to resolve this.

Jim Dempsey