MPI hangs on intranode communication

[LPN2024]

I am running a fortran+MPI job on a single server (Xeon(R) Gold 6430, 2 sockets, 32 cores per socket, hyperthreading on). The job:   

        1) pid=0: read some data

         2) broadcast data to all processors (not huge, < 150 MB)

         3) each process computes some statistics

         4) each process sends results to pid0

If I launch the job using 32 cores, the job runs quickly; everything works fine.

If I launch requesting 64 cores, the job hangs at step 2.

mpirun -np 32 build/mpi/SparseDemand.exe input.prop

mpirun -np 64 build/mpi/SparseDemand.exe input.prop

I have tried various combinations of I_MPI environment variables and tried tuning, but nothing has improved the situation.

Any thoughts on how to overcome this problem?

Thank you.

[TobiasK]

@LPN2024  please provide the full output of 

I_MPI_DEBUG=10 I_MPI_HYDRA_DEBUG=1 mpirun -np 64 IMB-MPI1

[LPN2024]

attached is the output of the benchmark

[TobiasK]

@LPN2024 

Since the output seems to be fine, the problem is likely related to your application.

If you develop the application by yourself, you may try to run it with a suitable debugger, and check if everything works as expected. If you do not develop this application, please ask the developers for guidance.

[LPN2024]

Ok, thank you, I will recheck the application in more detail.