MKL related calls not allowed inside offload regions?

[PKM]

Hi

I am implementing offload of an existing Fortran openMP region to Xeon Phi and have run into a few issues ...

I run the latest version of parallel studio 2013 with visual studio 2012 on Windows server 2008 R2.

1: Specifically, I call mkl_set_num_threads, mkl_set_dynamic, ZGETRS and ZGETRF inside my offload region and all of them return a runtime error:

On the sink, dlopen() returned NULL. The result of dlerror() is "/var/volatile/tmp/coi_procs/1/52855/load_lib/AarhusInv64IdebugMIC.out: undefined symbol: mkl_set_dynamic_"

2: Will offload of Fortran derived types including allocatable variables and type bound procedures be supported in the up-coming compiler version 15? This limitation really puts restrictions on the types of offload scenarios feasible ... :-(

Thanks!

C

[Kevin_D_Intel]

The calls should be allowed in the offload code. Within the scope you call them, have you tried adding:

 !DIR$ ATTRIBUTES offload:mic ::   mkl_set_num_threads, mkl_set_dynamic, ZGETRS, ZGETRF

We do not support offloading of a derived-type with an allocatable component and type bound procedures in the current release. Currently the derived-type must be bitwise-copyable without any allocatable component, so all fields must be numeric.  We are only adding support for the derived-type with an allocatable component in our next (15.0) release due out later this year; however, the support is only for a numeric allocatable component meaning it will not support nested allocations.

FWIW, where one can avoid offloading of the derived-type data and re-create the data on the coprocessor, you can use the full features of Fortran Standard the compiler supports within the offload code itself.

[PKM]

Thanks Kevin

I will try to write routines for disassembling the arrays in my derived types and re-assembling them again on the Phi. How about type-bound procedures? Are they supported in 15.0?

Regarding the mkl problem I have made the tiny code example below which follows your suggestion, but still reproduces the problem.

  program Console1
    implicit none
!DIR$ ATTRIBUTES offload:mic :: mkl_set_num_threads
     
!DIR$ OFFLOAD BEGIN TARGET(mic:0)
    call mkl_set_num_threads(100)
 !DIR$ END OFFLOAD   

    end program Console1

Compiler command line options:

/nologo /debug:full /Od /warn:interfaces /module:"x64\Debug\\" /object:"x64\Debug\\" /Fd"x64\Debug\vc110.pdb" /traceback /check:bounds /check:stack /libs:dll /threads /dbglibs /Qmkl:parallel /c

Linker command line options:

/OUT:"x64\Debug\Console1.exe" /INCREMENTAL:NO /NOLOGO /qoffload-ldopts="-lifcoremt" /MANIFEST /MANIFESTFILE:"c:\users\casper.kirkegaard\documents\visual studio 2012\Projects\Console1\Console1\x64\Debug\Console1.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"c:\users\casper.kirkegaard\documents\visual studio 2012\Projects\Console1\Console1\x64\Debug\Console1.pdb" /SUBSYSTEM:CONSOLE /IMPLIB:"c:\users\casper.kirkegaard\documents\visual studio 2012\Projects\Console1\Console1\x64\Debug\Console1.lib"

[Kevin_D_Intel]

There may be other ways to accomplish this and I will check an MKL colleague but you can add include "mkl.fi" after the implicit none statement and that should compile and run.

I’m not aware of any plans to support using type bound procedures with offloading. What sort of usage were you interested in?

[PKM]

Hi again Kevin

Adding the include statement as below still doesn't resolve the problem ....

C

    program Console1
    implicit none
    include 'mkl.fi'
!DIR$ ATTRIBUTES offload:mic :: mkl_set_num_threads
     
    !DIR$ OFFLOAD BEGIN TARGET(mic:0)
    call mkl_set_num_threads(100)  
    !DIR$ END OFFLOAD   

    end program Console1

[Kevin_D_Intel]

We have a problem with our IDE integrations. What I suggested (and tested) works via the command-line (e.g. ifort u518187.f90 /Qmkl:parallel), but the IDE integrations have a flaw where the /Qmkl setting (from Configuration Properties > Fortran > Libraries > Use Intel Math Kernel Library) is not propagated to the MIC-side link accordingly.

Here's a work around for now when using the IDE. Under Configuration Properties > Linker > General > Additional Options for MIC Offload Linker add the value: -mkl=parallel

Unfortunately, should you select a different setting under Configuration Properties > Fortran > Libraries > Use Intel Math Kernel Library later on,  you need to remember to mirror that setting change with the Linux option form under this Linker option too.

I opened a defect report with Development (see internal tracking id below) and will keep you posted on a fix. Our apologies for the inconvenience/confusion.

(Internal tracking id: DPD200358815)

[PKM]

Thanks Kevin - almost there now :-)

Adding that additional linker option now produces the following error:

The remote process indicated that the following libraries could not be loaded:

libmkl_intel_lp64.so libmkl_intel_thread.so libmkl_core.so

offload error: cannot load library to the device 0 (error code 19)

[PKM]

Oops .. Forgot to mention that i have obviously tried adding c:\Program Files (x86)\Intel\Composer XE 2013 SP1\mkl\lib\mic\ to linker->general->additional library directories ...

[Kevin_D_Intel]

The error indicates the environment variable setting for MIC_LD_LIBRARY_PATH is lacking the path to the MKL mic RTLs. By default this variable should be pre-initialized to include paths like this:

C:\>echo %MIC_LD_LIBRARY_PATH%
C:\Program Files (x86)\Intel\Composer XE 2013 SP1.176\compiler\lib\mic;C:\Program Files (x86)\Intel\Composer XE 2013 SP1.176\mkl\lib\mic;C:\Program Files (x86)\Intel\Composer XE 2013 SP1.176\compiler\lib\mic;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\compiler\lib\mic

Can you please check this environment variable's setting on your system?

[PKM]

I checked and it only includes the very last of those paths. I tried to set the env variable to match yours on the commandline before running the program and that doesn't work. Is it required to be set at link time? If so, how do i define the paths nicely? In windows system properties->environment variables it is set to MIC_LD_LIBRARY_PATH=%INTEL_DEV_REDIST%COMPILER\LIB\MIC ...

[Kevin_D_Intel]

I'm not sure yet why you are having this problem. There no additional user action necessary that I'm aware of to make this work. I have the same system properties setting as you showed.

Under the pre-initialized Intel compiler command-prompt this variable is setup via the compilervars scripts.

Under the IDE, within the small test case you provided earlier, I added code to fetch and print the setting which shows this:

character*256 MICpath

call GETENV ("MIC_LD_LIBRARY_PATH",MICpath)
print *, "MIC_LD_LIBRARY_PATH =",MICpath

Which displays:

MIC_LD_LIBRARY_PATH =
 C:\Program Files (x86)\Intel\Composer XE 2015\compiler\lib\mic;C:\Program Files
  (x86)\Intel\Composer XE 2015\mkl\lib\mic;

Press any key to continue . . .

I need to inquire w/others to learn how this is setup under the IDE. Also, I currently have the Beta compiler/integrations installed on my system so I need to perhaps revert to the previous 14.0 integrations you might using to see whether there is something awry with those that I am not seeing.

[Kevin_D_Intel]

Ok sorry, you must have CXE 2013 SP1 installed. I forgot about the older issue discussed this article. My advice is to upgrade to the latest update. The work around is only usable for the Debug configuration.

PS - An MKL colleague confirmed my earlier advice for using mkl.fi w/the ATTRIBUTE was correct and pointed out the sgemm.f90 example uses this same method.