Zheyu from Microsoft here. During our internal testing of Intel MPI on HPC Clusters using OSU benchmarks, we found that Intel MPI would often get stuck on MPI_Init(). The symptom would be that one of the threads would have a stack trace like the following:
while other threads would have a stack trace like the following:
Upon further extensive testing we found that the problem persists across multiple IMPI versions, UCX versions and other environment configurations. However, once we set I_MPI_STARTUP_MODE to pmi_shm, the hang goes away. This workaround, according to the documentation, comes with a performance cost for process startup.
We are suspecting that the bugfix mentioned in version 14.2 release notes, i.e. fixing of race condition in collectives causing hangs, is not properly applied to the collectives used in the startup process involved in the startup code path using netmod infrastructure (e.g. MPIDU_bc_allgather), causing the issue that we are witnessing. Could you investigate whether this is the case, or is this some other sort of race condition? What would be the planned release version for the fix?
Link Copied
