topic Re:Invalid argument errors using level zero IPC (zeMemOpenIpcHandle) in GPU Compute Software

Invalid argument errors using level zero IPC (zeMemOpenIpcHandle)

brucechen057 — Tue, 20 Jun 2023 16:47:40 GMT

I have 2 process, 2 GPUs and I want to use level zero IPC to transfer data. I follow the sample code in https://spec.oneapi.io/level-zero/latest/core/PROG.html#memory-1 . I send the IPC handle through MPI send/recv. To be specific, I create a IPC handle in process A with zeMemGetIpcHandle and sent the handle (either using MPI send/recv) to process B and open the handle in process B with zeMemOpenIpcHandle. However, I get the return code ZE_RESULT_ERROR_INVALID_ARGUMENT in zeMemOpenIpcHandle.

I have check that the values in the IPC handle are the same in process A and B. I also found that if I get and open the handle in only 1 process and I won't get the error.

Re:Invalid argument errors using level zero IPC (zeMemOpenIpcHandle)

AishwaryaCV_Intel — Wed, 21 Jun 2023 09:43:16 GMT

Hi,

Thank you for posting in intel communities.

Could you please provide the following details:

OS details
Hardware details
MPI version
Steps to reproduce the issue

Thanks And Regards,

Aishwarya

Re: Re:Invalid argument errors using level zero IPC (zeMemOpenIpcHandle)

brucechen057 — Wed, 21 Jun 2023 19:59:35 GMT

OS: Ubuntu 22.04.2 LTS

Hardware:
clinfo -l
Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM)
`-- Device #0: Intel(R) FPGA Emulation Device
Platform #1: Intel(R) OpenCL
`-- Device #0: Intel(R) Xeon(R) Platinum 8480+
Platform #2: Intel(R) OpenCL HD Graphics
+-- Device #0: Intel(R) Data Center GPU Max 1550
+-- Device #1: Intel(R) Data Center GPU Max 1550
+-- Device #2: Intel(R) Data Center GPU Max 1550
`-- Device #3: Intel(R) Data Center GPU Max 1550

MPI version: Intel(R) MPI Library for Linux* OS, Version 2021.9 Build 20230307 (id: d82b3071db)

Steps to reproduce the issue:

Use the attached example code, compile with `mpicc example.c -lze_loader`, and run with `UCX_TLS=ud,sm,self mpirun -np 2 ./a.out`

Get error at line 87