we would like to bring to your attention that the Intel MPI Library runtime (we tried both versions 2021.5 and 2021.6) produces a crash when mpiexec.hydra is ran. The crash happens in the following conditions:
- Operating System: CentOS 7, RHEL 7, RHEL 8, Oracle Linux 7 and Oracle Linux 8
- the environment variable I_MPI_HYDRA_TOPOLIB is set to "ipl"
- the crash occurs regardless of the number of processes (-np)
- the crash occurs with any executable
Note: on the same virtual machines, using the Intel MPI runtime 2018.4 is working fine.
In the two archives attached to this thread, you can find the evidences for Oracle Linux 7 and Oracle Linux 8. More in detail, each .tar.gz file contains the following:
- hwloc-ls.txt: output from the hwloc-ls command
- lstopo.xml: the hwloc topology exported in XML format (so you can import it to reproduce in your lab)
- lscpu.txt: output from the lscpu command
- mpiexec-hydra-valgrind.txt: output of the mpiexec.hydra with the environment variable HYDRA_BSTRAP_VALGRIND=1
- vgcore.XXXXXX: core dump produced by Valgrind after the crash
Note: the reason why we would like to use IPL as topology library is because the hwloc implementation seems to be buggy and not reporting the correct CPU topology. This leads to incorrect process pinning, where all processes are assigned to CPU #0.
Many thanks in advance.
Thank you for posting in Intel Communities.
We were able to reproduce your issue at our end using the Intel MPI Library 2021.6 on a Centos 7 machine as shown in below screenshot:
We are working on your issue and will get back to you soon.
Thanks & Regards,