Problems creating more than one _Offload_stream stream

manumachu__ravi · ‎06-02-2016

Dear MIC Developers,

I am using Offload Streams explained in the page below and interface described in the include file offload.h.

https://software.intel.com/en-us/node/583657
/opt/mkl/compilers_and_libraries_2016.2.181/linux/compiler/include/offload.h

I am not able to create more than one _Offload_stream as follows:

    _Offload_stream sHandle1 = _Offload_stream_create(0, 1);
    _Offload_stream sHandle2 = _Offload_stream_create(0, 1);
    /* Some computations and data transfers */
    _Offload_stream_destroy(0, sHandle1);
    _Offload_stream_destroy(0, sHandle2);

During the creation of the second stream "sHandle2", I am seeing the following error:
offload_error: unable to set affinity

Could you please tell me if I can not create more than one offload stream at a time?

Do you suggest I use 'libxstream' for this purpose?
https://github.com/hfp/libxstream

Regards
Ravi

My experimental machine has 2 sockets with 12 physical cores each (Intel Haswell Processors). 'micinfo' displays the following information about the host and the coprocessor.

$micinfo
MicInfo Utility Log
Created Thu Jun 2 21:34:44 2016

        System Info
                HOST OS                 : Linux
                OS Version              : 3.10.0-327.13.1.el7.x86_64
                Driver Version          : 3.6.1-1
                MPSS Version            : 3.6.1

Host Physical Memory : 64152 MB

Device No: 0, Device Name: mic0

        Version
                Flash Version            : 2.1.02.0391
                SMC Firmware Version     : 1.17.6900
                SMC Boot Loader Version : 1.8.4326
                Coprocessor OS Version   : 2.6.38.8+mpss3.6.1
                Device Serial Number     : ADKC51700518

        Board
                Vendor ID                : 0x8086
                Device ID                : 0x225d
                Subsystem ID             : 0x3604
                Coprocessor Stepping ID : 2
                PCIe Width               : Insufficient Privileges
                PCIe Speed               : Insufficient Privileges
                PCIe Max payload size    : Insufficient Privileges
                PCIe Max read req size   : Insufficient Privileges
                Coprocessor Model        : 0x01
                Coprocessor Model Ext    : 0x00
                Coprocessor Type         : 0x00
                Coprocessor Family       : 0x0b
                Coprocessor Family Ext   : 0x00
                Coprocessor Stepping     : C0
                Board SKU                : C0PRQ-3120/3140 P/A
                ECC Mode                 : Enabled
                SMC HW Revision          : Product 300W Passive CS

        Cores
                Total No of Active Cores : 57
                Voltage                  : 0 uV
                Frequency                : 1100000 kHz

        Thermal
                Fan Speed Control        : N/A
                Fan RPM                  : N/A
                Fan PWM                  : N/A
                Die Temp                 : 41 C

        GDDR
                GDDR Vendor              : Elpida
                GDDR Version             : 0x1
                GDDR Density             : 2048 Mb
                GDDR Size                : 5952 MB
                GDDR Technology          : GDDR5
                GDDR Speed               : 5.000000 GT/s
                GDDR Frequency           : 2500000 kHz
                GDDR Voltage             : 1501000 uV

Rajiv_D_Intel · ‎06-02-2016

At present when using streams it is necessary to set KMP_AFFINITY and OFFLOAD_STREAM_AFFINITY.

For example:

export MIC_ENV_PREFIX=MIC
export MIC_KMP_AFFINITY="norespect,none"
export OFFLOAD_STREAM_AFFINITY=compact

manumachu__ravi · ‎06-02-2016

Thank you, Rajiv.

Could I please ask you if there is a manual for this functionality (Offload streams)? The article "https://software.intel.com/en-us/node/583657" seems to be the only source.

Best Regards
Ravi

jimdempseyatthecove · ‎06-06-2016

Ravi,

Part of the design of the offload system is to fall back onto the host processor should the (designated) MIC unavailable. Does the offload stream follow this capability? In particular, will streams code developed on KNC just work on KNL?

Jim Dempsey

Rajiv_D_Intel · ‎06-06-2016

If stream creation using _Offload_stream_create is successful then offloads to that stream will succeed.

Streams code that works on KNC should work just the same on KNL.