Offload from dynamically loaded library ?

[William_H_3]

Hello,

I am new to programming for the Xeon Phi, and am having trouble getting explicit offload to work when the offload is done in a dynamically loaded library. The code fails at run time at the offload location, with the message

"offload error: cannot load library to the device 0 (error code 5)"

I am using C on  Scientific Linux 7, using Parallel Studio XE 2015 Update 1 Cluster Edition.  

A simple example of what I'm trying to do is below :

A basic main program which loads the dynamic library and calls the function in it :

#include <dlfcn.h>

int main()

{

        int (*fn)();
        
        void *lib_handle;
        lib_handle = dlopen("./hello.so", RTLD_LAZY);
 
        fn = dlsym(lib_handle, "sayhello");
        (fn)();
}        

This is compiled with :

icc loadsharedhello.c

And the contents of dynamic library:

#include <stdio.h>

 __attribute__((target(mic))) void MyFunction() {
         printf("Hello World from coprocessor!\n");
        fflush(0);
 }

 int sayhello () {
         printf("Hello World from host!\n");
 #pragma offload target(mic)
         {
                 MyFunction();
         }
                printf("Bye\n");
 }

This is compiled with:

icc  -shared -fpic dynamichello.c -o hello.so               

If I do the offload in the main program, it works fine. Any clues as to what I'm doing wrong ?

Thanks,

Bill

[William_H_3]

I forgot to add that if I set the environment variable OFFLOAD_REPORT=3,

I get an additional line in the error message :

[Offload] [HOST]          [State]           Initialize logical card 0 = physical card 0
offload error: cannot load library to the device 0 (error code 5)

[pbkenned1]

I don't think you've coded anything incorrectly.  What version of MPSS are you using?

Perhaps there's an issue with your OS -- officially Scientific Linux 7 is not supported for Phi systems, although other non-official OSes can be made to work, eg, Ubuntu 12.04 or later. 

Is Scientific Linux 7 based on Redhat 7? 

[U537432]$ icc -V
Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.1.133 Build 20141023
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

[U537432]$ icc  -shared -fpic dynamichello.c -o hello.so
[U537432]$ icc loadsharedhello.c -o loadsharedhello.x
[U537432]$ ./loadsharedhello.x
Hello World from host!
Bye
Hello World from coprocessor!
[U537432]$

Patrick

[William_H_3]

mpss is version 3.4.2

Yes, Scientific Linux 7 is based on Red Hat 7

Thanks for the reply.

Bill

[pbkenned1]

RHEL 7 + MPSS 3.4.2 is a supported combination, so I think there is a reasonable chance this can be made to work with Scientific Linux 7.  Did you doing anything special (like build MPSS from source), or did you just install the compilers and MPSS right out of the box 'as-is'?

Can you run miccheck to verify you're system is functioning properly?  The following is from an older MPSS system, but you should see something like the following if the system is OK:

[root~]# miccheck
MicCheck 3.2-r1
Copyright 2013 Intel Corporation All Rights Reserved

Executing default tests for host
  Test 0: Check number of devices the OS sees in the system ... pass
  Test 1: Check mic driver is loaded ... pass
  Test 2: Check number of devices driver sees in the system ... pass
  Test 3: Check mpssd daemon is running ... pass
Executing default tests for device: 0
  Test 4 (mic0): Check device is in online state and its postcode is FF ... pass
  Test 5 (mic0): Check ras daemon is available in device ... pass
  Test 6 (mic0): Check running flash version is correct ... pass
Executing default tests for device: 1
  Test 7 (mic1): Check device is in online state and its postcode is FF ... pass
  Test 8 (mic1): Check ras daemon is available in device ... pass
  Test 9 (mic1): Check running flash version is correct ... pass

Status: OK
[root~]#

Patrick

[pbkenned1]

A colleague of mine also made this suggestion (he's unable to access this Forum at the moment) --

In your dlopen() call, use the full pathname to the .so instead of a relative pathname.

Patrick

[William_H_3]

Hi,

Thanks again for the speedy replies.  Output of miccheck looks good to me:

MicCheck 3.4.2-r1
Copyright 2013 Intel Corporation All Rights Reserved

Executing default tests for host
  Test 0: Check number of devices the OS sees in the system ... pass
  Test 1: Check mic driver is loaded ... pass
  Test 2: Check number of devices driver sees in the system ... pass
  Test 3: Check mpssd daemon is running ... pass
Executing default tests for device: 0
  Test 4 (mic0): Check device is in online state and its postcode is FF ... pass
  Test 5 (mic0): Check ras daemon is available in device ... pass
  Test 6 (mic0): Check running flash version is correct ... pass
  Test 7 (mic0): Check running SMC firmware version is correct ... pass

Status: OK

The use of an absolute path sounded promising to me, but didn't have any effect.  

Bill

[Frances_R_Intel]

I wonder if it has anything to do with using dlopen and dlsym rather than linking to the hello.so library when you build the executable. Do the necessary C libraries used by hello.so get copied to the coprocessor when the linking is done using dlopen? Perhaps the machine Patrick is using has the libraries permanently installed or NFS mounted to the coprocessor but the machine William is using does not.

[Kevin_D_Intel · Accepted solution]

The error is reproducible under MPSS 3.4.1 only with our 15.0 release. The error differs slightly depending on whether one uses the 15.0 initial release or Update 1 release. The initial release throws "offload error: target executable is not available". Update 1 throws the error you showed.

With 15.0 when the main program lacks offload but uses .so that do, you must add -offload to at least the linking of the main program.

To resolve your issue, add -offload to compilation (or the link if you want to compile the main to object first) for the main program and also ensure your MIC_LD_LIBRARY_PATH environment variable setting includes the path to the hello.so. That can either be “.” or if you prefer the full path on the host where hello.so resides. It does not matter whether you include "./" in the dlopen or not since that path is only relevant on the host where dlopen executes.

$ icc -shared -fpic dynamichello.c -o hello.so
$ icc loadsharedhello.c -c
$ icc loadsharedhello.o
$ ./a.out
Hello World from host!
offload error: cannot load library to the device 0 (error code 5)

$ icc loadsharedhello.o -offload
$ ./a.out
Hello World from host!
offload error: cannot load library to the device 0 (error code 5)

$ export MIC_LD_LIBRARY_PATH=.:$MIC_LD_LIBRARY_PATH
$ ./a.out
Hello World from host!
Bye
Hello World from coprocessor!

[William_H_3]

Excellent ! That resolves the problem.  Thanks very much to all who responded for your attention to this issue.

Bill

[Minh_H_]

Hi,

System: parallel_studio_xe_2015, Centos 7.0, MPSS 3.4.2.

I got similar  problem but the above solution does not work.

I created a dynamic library   .so, which will load from python. I did add -offload option when make the library. The library will do main calculation, which also offloads to Xeon Phi.

I also did

export MIC_LD_LIBRARY_PATH=/home/beowulf/intel:/opt/intel/composer_xe_2015.0.090/compiler/lib/mic:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/amplxe/device/lib64

but when I run the python code, which will load the dynamic library. It reports the following error

"offload error: target executable is not available"

However  I can run the offload program directly from the command line.

Any advice to solve this problem?

I also tried the tricks in
https://software.intel.com/en-us/forums/topic/534589

However due to different compiler version, I could not find similar locations for all paths.

[Kevin_D_Intel]

What MPSS version are you using?

The -offload is used for compilation/link of the main program that calls into a .so (containing offload code) but where the main program itself does not include any offload code. The option has no effect building the .so  that already contains offload code.

Ensure the MIC_LD_LIBRARY_PATH setting includes the path to where your offloaded .so resides. I’m thinking perhaps you have via the /home path you included.

I worked on something similar to what you described a few releases back but I do not have a mockup handy at the moment that I can tinker with but will try creating one to see what is required with the newer compiler/MPSS releases.

[Minh_H_]

Hi Kevin,

MPSS 3.4.2.

I did include the path to .so in MIC_LD_LIBRARY_PATH as you realized (/home/beowulf/intel).

Please let me know when you have a solution for this problem.

Thanks, M

[Minh_H_]

Hi Kevin,

System: parallel_studio_xe_2015, Centos 7.0, MPSS 3.4.2.

Here are simple python interface code could reproduce the problem

-----------------------------------------------------------------------------------------

/*
 * Create simple python interface
 * File: pymichello.c
 * compile:
 *  icc   -shared  -O2 -fPIC  -offload-attribute-target=mic  -I /usr/include/python2.7  -I /usr/lib64/python2.7/site-packages/numpy/core/include   pymichello.c  -openmp -lc   -o _pymichello.so
 *  
 * */

#include "Python.h"
#include "numpy/arrayobject.h"
#include <stdio.h>

static char  py_michello_doc[] = "Hello from MIC ";

static PyObject *py_michello(PyObject *self, PyObject *args)
{
#pragma offload target(mic)
    {
        printf("Hello from MIC");
    }
    
    printf("Hello from host");
    return Py_BuildValue("I",0);
}

static PyObject *py_michello1(PyObject *self, PyObject *args)
{
    int i1, i2;
    i1 = 1;
    i2 = 2;
#pragma offload target(mic) inout(i1)
    {
        i1 = i1*224;
    }
    
                
    return Py_BuildValue("I",i1+ i2);
}

static  PyMethodDef  _pymichellomethods[] =
{

        {"py_michello", py_michello, METH_VARARGS, py_michello_doc},
        {"py_michello1", py_michello1, METH_VARARGS, py_michello_doc},
        {NULL, NULL, 0, NULL}
};

/* Python 2 module initialisation */
//add module clean up using:  int Py_AtExit(void (*func) ()): register clean up function
void init_pymichello(void)
{
    PyObject *mod;
    mod = Py_InitModule("_pymichello",_pymichellomethods);
    import_array();
    //clean up module
    //Py_AtExit(py_module_clean);
}

----------------------------------------------------------------------------

#File : michello.py

from _pymichello  import  py_michello, py_michello1

print(py_michello1())
py_michello()

---------------------------------------------

 export OFFLOAD_INIT=

offload error: target executable is not available

-----------------------------------------------------

export OFFLOAD_INIT=on_start

[Offload] [MIC 0] [File]                    pymichello.c
[Offload] [MIC 0] [Line]                    26
[Offload] [MIC 0] [Tag]                     Tag 0
[Offload] [HOST]  [Tag 0] [State]           Start target
[Offload] [HOST]  [Tag 0] [State]           Setup target entry: __offload_entry_pymichello_c_26py_michello1icc1406040670nttqy4
[Offload] [HOST]  [Tag 0] [State]           Host->target pointer data 0
[Offload] [HOST]  [Tag 0] [Signal]          signal : none
[Offload] [HOST]  [Tag 0] [Signal]          waits  : none
[Offload] [HOST]  [Tag 0] [State]           Host->target pointer data 0
[Offload] [HOST]  [Tag 0] [State]           Host->target copyin data 4
[Offload] [HOST]  [Tag 0] [State]           Execute task on target
offload error: cannot create pipeline on the device 0 (error code 14)

[Kevin_D_Intel]

Thank you for the test case. That was very helpful and useful.

There are two issues at play, a compiler and an MPSS 3.4.1. I do not have a complete resolution yet when using the MPSS 3.4.1 release nor know whether 3.4.2 has a fix so I have to upgrade and test MPSS 3.4.2 also.

To resolve the issue/error you currently experienced you will need the IPS XE 2015 Update 1 release at least so you will have to upgrade your IPS release. When using that release the issue with MPSS 3.4.1 surfaces which I am still investigating. Some good news is that I confirmed that your test case and one I worked earlier both run successfully using the IPS XE 2015 Update 1 release but on the earlier MPSS 3.2.4 release only. With MPSS 3.4.1 there is a run-time issue with loading dependent .so libs. I will keep you updated as I learn more.

[Minh_H_]

Hi Kevin,

Does the update IPS XE 2015 Update 1 work for MPSS 3.4.2?  Do I need to downgrade to MPSS 3.2.4 to be able to offload code from dynamic library? Is this safe to do so, i.e., re-flash with older  flash version?

  Thanks

[Kevin_D_Intel]

Yes, IPS XE 2015 Update 1 is compatible with the earlier MPSS 3.2.4 release (which I think is what you were asking). Unfortunately, if you need to progress then downgrading to MPSS 3.2.4 is really the only option since I cannot offer any alternative/more convenient work around at the moment. The MPSS may be looking into this issue shortly and might have a more convenient work around if you are able to wait.

In downgrading MPSS, if it were me, I would initially just uninstall MPSS 3.4.2 and install MPSS 3.2.4 and *not* re-flash and check whether offload works. Personally, I would avoid re-flashing to older flash, BUT, I say that without any experience/knowledge whether it is problematic or safe. I have just avoided ever re-flashing to older flash.

Updated: I filed the issue with the MPSS Development team. (Internal tracking id: HSD #5162051)

(Resolution Update on 04/08/2015): This defect is fixed the MPSS 3.5 release

[Kevin_D_Intel]

The fix for the earlier noted HSD #5162051 will be in the upcoming MPSS 3.5 release. I don't have a definitive release date for this new MPSS release. I'll let you know once I hear. You can also subscribe to the Intel® Manycore Platform Software Stack (MPSS) page for updates.

[Kevin_D_Intel]

The MPSS 3.5 release was just announced. You can refer to the Intel® Manycore Platform Software Stack (MPSS) page for some details and links to the MPSS 3.5 downloads. While I verified a fix internally for the earlier indicated HSD (the loading of dependent .so libs issue), I have not yet installed MPSS 3.5 to verify the fix using the actual release, but it should be included.