topic MPI hangs on intranode communication in Intel® MPI Library

MPI hangs on intranode communication

LPN2024 — Thu, 31 Oct 2024 14:17:23 GMT

I am running a fortran+MPI job on a single server (Xeon(R) Gold 6430, 2 sockets, 32 cores per socket, hyperthreading on). The job:

1) pid=0: read some data

2) broadcast data to all processors (not huge, < 150 MB)

3) each process computes some statistics

4) each process sends results to pid0

If I launch the job using 32 cores, the job runs quickly; everything works fine.

If I launch requesting 64 cores, the job hangs at step 2.

mpirun -np 32 build/mpi/SparseDemand.exe input.prop

mpirun -np 64 build/mpi/SparseDemand.exe input.prop

I have tried various combinations of I_MPI environment variables and tried tuning, but nothing has improved the situation.

Any thoughts on how to overcome this problem?

Thank you.

Re: MPI hangs on intranode communication

TobiasK — Tue, 05 Nov 2024 10:30:41 GMT

@LPN2024 please provide the full output of

I_MPI_DEBUG=10 I_MPI_HYDRA_DEBUG=1 mpirun -np 64 IMB-MPI1

Re: MPI hangs on intranode communication

LPN2024 — Tue, 05 Nov 2024 11:59:50 GMT

attached is the output of the benchmark

Re: MPI hangs on intranode communication

TobiasK — Tue, 05 Nov 2024 12:21:32 GMT

@LPN2024

Since the output seems to be fine, the problem is likely related to your application.

If you develop the application by yourself, you may try to run it with a suitable debugger, and check if everything works as expected. If you do not develop this application, please ask the developers for guidance.

Re: MPI hangs on intranode communication

LPN2024 — Tue, 05 Nov 2024 18:01:31 GMT

Ok, thank you, I will recheck the application in more detail.