topic Re:Intel oneAPI MPI runtime 2021.6 -- mpiexec.hydra crashes when I_MPI_HYDRA_TOPOLIB=ipl in Intel® MPI Library

Intel oneAPI MPI runtime 2021.6 -- mpiexec.hydra crashes when I_MPI_HYDRA_TOPOLIB=ipl

AlefRome — Sat, 21 May 2022 17:54:49 GMT

Hi All,

we would like to bring to your attention that the Intel MPI Library runtime (we tried both versions 2021.5 and 2021.6) produces a crash when mpiexec.hydra is ran. The crash happens in the following conditions:

Operating System: CentOS 7, RHEL 7, RHEL 8, Oracle Linux 7 and Oracle Linux 8
the environment variable I_MPI_HYDRA_TOPOLIB is set to "ipl"
the crash occurs regardless of the number of processes (-np)
the crash occurs with any executable

Note: on the same virtual machines, using the Intel MPI runtime 2018.4 is working fine.

In the two archives attached to this thread, you can find the evidences for Oracle Linux 7 and Oracle Linux 8. More in detail, each .tar.gz file contains the following:

hwloc-ls.txt: output from the hwloc-ls command
lstopo.xml: the hwloc topology exported in XML format (so you can import it to reproduce in your lab)
lscpu.txt: output from the lscpu command
mpiexec-hydra-valgrind.txt: output of the mpiexec.hydra with the environment variable HYDRA_BSTRAP_VALGRIND=1
vgcore.XXXXXX: core dump produced by Valgrind after the crash

Note: the reason why we would like to use IPL as topology library is because the hwloc implementation seems to be buggy and not reporting the correct CPU topology. This leads to incorrect process pinning, where all processes are assigned to CPU #0.

Many thanks in advance.

Best Regards.

Pietro.

Re: Intel oneAPI MPI runtime 2021.6 -- mpiexec.hydra crashes when I_MPI_HYDRA_TOPOLIB=ipl

SantoshY_Intel — Mon, 23 May 2022 12:12:39 GMT

Hi,

Thank you for posting in Intel Communities.

We were able to reproduce your issue at our end using the Intel MPI Library 2021.6 on a Centos 7 machine as shown in below screenshot:

We are working on your issue and will get back to you soon.

Thanks & Regards,

Santosh

Re:Intel oneAPI MPI runtime 2021.6 -- mpiexec.hydra crashes when I_MPI_HYDRA_TOPOLIB=ipl

SantoshY_Intel — Fri, 27 May 2022 11:05:09 GMT

Hi,

We have reported this issue to the concerned development team. They are looking into your issue.

Thanks & Regards,

Santosh

Re:Intel oneAPI MPI runtime 2021.6 -- mpiexec.hydra crashes when I_MPI_HYDRA_TOPOLIB=ipl

DrAmarpal_K_Intel — Tue, 12 Mar 2024 12:45:52 GMT

Hello! This issue has been fixed in the latest release of Intel MPI. hwloc (default) should also work as expected.

Re:Intel oneAPI MPI runtime 2021.6 -- mpiexec.hydra crashes when I_MPI_HYDRA_TOPOLIB=ipl

DrAmarpal_K_Intel — Tue, 12 Mar 2024 12:48:18 GMT

“This issue has been resolved and we will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread. Any further interaction in this thread will be considered community only”.