Data transfer between MIC in offload model

[H__Kamil]

Hi,

I would like to ask, if is it possible to echange data directly between two MICs in offload model?  My application belongs to forward-in-time groups of algorithms. Due to data dependencies, I have to transfer data between devices in every time step. I am using up to 5 Intel KNL processors as accelerators (offload over fabric) and in this moment host memory is a point of data exchange. In order to optimize path of data transfer, I thought about transferring data directly between MICs. However I do not know, how to achieve this goal using offloading programming model. 

Thanks for help.
Best regards,
Kamil :)
 

[Sebastian_S_Intel]

Dear Kamil,

unfortunately the functionality you are looking for does not exist in pragma offload programming model which is Copyin/Copyout model.

Regards
Sebastian

[jimdempseyatthecove]

Kamil,

Let me preference this by: I haven't done this myself....

I suggest you take a look at using the MPI messaging system. Depending on how the fabric is implemented amongst MICs you may have available direct MIC<->MIC transfer capability over the PCIe bus. You can blend OpenMP and OpenMPI if you desire to do so.

Jim Dempsey

[TimP]

It was my understanding that MPI wasn't necessarily more efficient for communication among KNC nodes unless you have IB cards installed, as the data would be copied through host when using either your offload system or MPI. 

As to current and future MIC products, it does seem that MPI (with linux) is the primary method.

[Jan_Z_Intel]

jimdempseyatthecove wrote:

[...] you may have available direct MIC<->MIC transfer capability over the PCIe bus. 

Jim,

Kamil is using offloading over fabric, which means that he is offloading using compiler/OpenMP pragmas to Intel(R) XeonPhi(TM) processor based server connected to his host via fabric network, not to a coprocessor attached via PCI. This a different hardware configuration. 

Kamil,

Do you have an existing code base or are you creating your software from scratch?

Best regards,
Jan 

[H__Kamil]

Dear Jan,

I have the stable version of my application which works well on hybrid platform with two KNC coprocessors.  Now, based on this code I would like to build practically new code for hybird platforms equipped with up to 6 KNL processors connected to host via fabric. 

Best regards.