Using oneAPI buffers in MPI calls

[germainf]

Hello,
As far as I understand, the buffer object is an abstract view of data.
Then, an accessor is needed (with an access mode) to interact with that data.

Most of MPI implementation provide a way to use CUDA device pointers to perform MPI communication.

Is there a way to perform MPI calls with this data abstraction? (with IntelMPI? with OpenMPI?)
Does the oneAPI MKL provide parallel implementation through MPI?

Thanks

Florent Germain

[PrasanthD_intel]

Hi Florent,

Yes, we can use both MPI and DPCPP and write a hybrid application. 

I am attaching a sample code and Makefile on how to use DPCPP along with MPI.

You can use OneAPI MKL with dpcpp in which you can use MKL call to launch kernels. I am attaching a GitHub repo link, please refer to this link https://github.com/intel/HPCkit-code-samples

Let us know if you need anything.

Regarding how to use MKL with IMPI I will get back to you.

Regards

Prasanth

[germainf]

Hi,

Thanks for the answer and the examples, it will help a lot.

Regards

Florent

[germainf]

Hi,

An error occurs when I try to run an MPI programm on devcloud:

mpirun -n 2 bin/vector_copy
Abort(2664079) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(136).........:
MPID_Init(1127)...............:
MPIDI_SHMI_mpi_init_hook(29)..:
MPIDI_POSIX_mpi_init_hook(141):
MPIDI_POSIX_eager_init(2109)..:
MPIDU_shm_seg_commit(261).....: unable to allocate shared memory

This seems to be an communication layer initialization failure. Is there any way to run an mpi based programm on the oneapi devcloud? I attach a copy of my source code and Makefile.

Regards

Florent

[PrasanthD_intel · Accepted solution]

Hi Florent,

Sorry for the late reply.

After logging into the devcloud you will be in a login node. You have to launch MPI in a compute node through a queue instead on a login node.

To run in a compute node, you must submit a job through a queue.

You can submit jobs to the queue using below command (Here you can change the nodes as per requirement, below command will allocate 2 nodes)

qsub -I -l nodes=2:ppn=2

You can see the nodefile(echo $PBS_NODEFILE) to check which nodes are allocated to you.

Let us know if you need any help

Regards

Prasanth

[PrasanthD_intel]

Hi Florent,

Since you have marked my response as a solution, I think you are able to use IMPI and DPCPP together.

As I said I will get back to you on how to run IMPI, DPCPP and OneMKL together, here is the command below. You must link MPI and MKL libraries with dpcpp.

dpcpp -g -o<executable> <Foo.cpp> -I$MKLROOT/include $MKLROOT/lib/intel64/libmkl_sycl.a -L$MKLROOT/lib/intel64 -I$I_MPI_ROOT/include/ -L$I_MPI_ROOT/lib/release -DMKL_ILP64 -lmpi -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lOpenCL -ldl

Let us know if you face any issues.

Regards

Prasanth

[PrasanthD_intel]

Hi Florent,

We haven't heard back from you. If you face any issue while using those commands, please reach out to us.

Else if you found the answer helpful please confirm so we can close this thread.

Regards

Prasanth

[germainf]

Hi,

I posted a new topic here: https://community.intel.com/t5/Intel-oneAPI-Base-Toolkit/Use-mathimf-h-with-dpcpp/td-p/1215586 with the new issue I encountered to avoid this thread to diverges.

I think this thread can be closed, thanks for your help, it helped a lot.

Regards

Florent

[PrasanthD_intel]

Hi Florent,

Thanks for the confirmation. We are closing this thread. Any further interaction in this thread will be considered community only

Regards

Prasanth