topic Re: Intel mpi hangs when receive message sent from the same rank in Intel® MPI Library

Intel mpi hangs when receive message sent from the same rank

jellie_ — Fri, 05 Jan 2024 09:00:22 GMT

Suppose I have a mpi program which will send/receive within the same rank:

#include <iostream> #include <mpi.h> int main(int argc, char** argv) { MPI_Init(&argc, &argv); int i1 = 2; MPI_Request req1; MPI_Isend(&i1, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1); MPI_Status status1; MPI_Wait(&req1, &status1); std::cout << "sent\n"; int i2; MPI_Status status2; MPI_Recv(&i2, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &status2); std::cout << i2 << "\n"; MPI_Finalize(); }

run it with mpirun -np 1 <executable>

when linked to openmi, it works well. However, when compiled and link to intel mpi, it hangs after printed the "sent".

If it is link to openmpi and run with intel's mpirun, it works.

Initially I suspect that a mpi rank communicate with itself is a undefinded behaviour, but I found this on stackoverflow:

https://stackoverflow.com/questions/11385395/is-the-behavior-of-mpi-communication-of-a-rank-with-itself-well-defined

so this might be a bug related to intel mpi.

openmpi version:

$ mpirun --version mpirun (Open MPI) 4.1.4 Report bugs to http://www.open-mpi.org/community/help/

intel mpi version:

Re: Intel mpi hangs when receive message sent from the same rank

jellie_ — Mon, 08 Jan 2024 06:10:45 GMT

additional infomation:

$ ldd test linux-vdso.so.1 (0x00007ffe1bdf2000) libc++abi.so.1 => /lib/x86_64-linux-gnu/libc++abi.so.1 (0x00007febf62f2000) libmpi.so.12 => /opt/intel/oneapi/mpi/2021.11/lib/libmpi.so.12 (0x00007febf4600000) libc++.so.1 => /lib/x86_64-linux-gnu/libc++.so.1 (0x00007febf61ec000) libunwind.so.1 => /lib/x86_64-linux-gnu/libunwind.so.1 (0x00007febf61dd000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007febf4521000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007febf61bd000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007febf4340000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007febf61b8000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007febf61b1000) /lib64/ld-linux-x86-64.so.2 (0x00007febf635a000)

Re: Intel mpi hangs when receive message sent from the same rank

jellie_ — Mon, 08 Jan 2024 07:53:04 GMT

another example：

#include <iostream> #include <mpi.h> int main(int argc, char** argv) { MPI_Init(&argc, &argv); int i1 = 2; MPI_Request req1; MPI_Isend(&i1, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1); MPI_Status status1; MPI_Wait(&req1, &status1); std::cout << "sent\n"; int i2; MPI_Status status2; while (true) { MPI_Iprobe(0, 0, MPI_COMM_WORLD, &i2, &status2); std::cout << i2 << '\n'; if (i2 != 0) { break; } } std::cout << "iprobe success\n"; MPI_Finalize(); }

when linked to openmpi, it ends normally.

when linked to intel mpi, the MPI_Iprobe never success, and the line 19 always prints 0.

Re:Intel mpi hangs when receive message sent from the same rank

TobiasK — Thu, 11 Jan 2024 14:58:44 GMT

@jellie_

both of your codes work for me, can you please add I_MPI_DEBUG=10 mpirun ... ?

Re: Re:Intel mpi hangs when receive message sent from the same rank

jellie_ — Thu, 11 Jan 2024 15:11:41 GMT

hello,

the output of the first example looks like this:

$ I_MPI_DEBUG=10 mpirun -np 1 executable [0] MPI startup(): Intel(R) MPI Library, Version 2021.11 Build 20231005 (id: 74c4a23) [0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric loaded: libfabric.so.1 [0] MPI startup(): libfabric version: 1.18.1-impi [0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1) [0] MPI startup(): libfabric provider: tcp [0] MPI startup(): File "" not found [0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.11/opt/mpi/etc/tuning_icx_shm-ofi.dat" [0] MPI startup(): File "/opt/intel/oneapi/mpi/2021.11/opt/mpi/etc/tuning_icx_shm-ofi.dat" not found [0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.11/opt/mpi/etc//tuning_clx-ap_shm-ofi.dat" [0] MPI startup(): threading: mode: direct [0] MPI startup(): threading: vcis: 1 [0] MPI startup(): threading: app_threads: -1 [0] MPI startup(): threading: runtime: generic [0] MPI startup(): threading: progress_threads: 0 [0] MPI startup(): threading: async_progress: 0 [0] MPI startup(): threading: lock_level: global [0] MPI startup(): tag bits available: 19 (TAG_UB value: 524287) [0] MPI startup(): source bits available: 20 (Maximal number of rank: 1048575) [0] MPI startup(): ===== Nic pinning on X1-Nano ===== [0] MPI startup(): Rank Pin nic [0] MPI startup(): 0 ve-ArchLinux [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 143295 X1-Nano {0,1,2,3,4,5,6,7} [0] MPI startup(): I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11 [0] MPI startup(): ONEAPI_ROOT=/opt/intel/oneapi [0] MPI startup(): I_MPI_MPIRUN=mpirun [0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=-1 [0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default [0] MPI startup(): I_MPI_DEBUG=10 sent

then it hangs

Re: Re:Intel mpi hangs when receive message sent from the same rank

TobiasK — Thu, 11 Jan 2024 15:22:44 GMT

Please make sure that you are running in a supported environment:

https://www.intel.com/content/www/us/en/developer/articles/system-requirements/mpi-library-system-requirements.html

Re: Re:Intel mpi hangs when receive message sent from the same rank

jellie_ — Thu, 11 Jan 2024 15:42:33 GMT

I'm using debian 12 (which is the last stable debian version), is this too new?

the clang 17 and gcc 12.2.0 compiler are all tested and the program always hangs.

Re: Re:Intel mpi hangs when receive message sent from the same rank

TobiasK — Fri, 12 Jan 2024 18:13:21 GMT

Debian 13 is currently not validated / supported.
On our machines, it works fine, however, the code itself is unsafe and also talking to my colleagues, we strongly recommend to avoid that as it may or may not work the way you want it to work.