topic Re: SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect in Intel® MPI Library

SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

menno — Mon, 21 Jun 2021 12:51:42 GMT

(repost from incorrect category)

A simulation package I'm working on uses repeated communications between participants by connecting, sending data, then disconnecting. This is not something I can change.

When I run this on Intel MPI (I tried versions 2020 Update 1 and 2), this gives a segmentation fault after several connect/send/disconnect cycles. The exact number varies from several tens to a few thousand.

I have created a simple reproduction program (see attached).

This works fine if I use OpenMPI 4.0.2 compiled with GCC 8.3.0 and compile the program with the mpicc from OpenMPI, but segfaults with Intel compiler and MPI.

It would be nice if this can be corroborated. And if so, how to solve the problem.

Re:SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

ShivaniK_Intel — Tue, 22 Jun 2021 12:14:07 GMT

Hi,

Thanks for reaching out to us.

We are able to reproduce the issue at our end. We are working on it and will get back to you soon.

Thanks & Regards

Shivani

Re:SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

ShivaniK_Intel — Wed, 30 Jun 2021 13:10:22 GMT

Hi,

Could you please provide the OFI provider you have been using?

Thanks & Regards

Shivani

Re:SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

ShivaniK_Intel — Wed, 07 Jul 2021 11:49:23 GMT

Hi,

As we didn't hear back from you, Is your issue resolved? If not, please provide the details that have been asked in my previous post.

Thanks & Regards

Shivani

Re:SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

Jennifer_D_Intel — Wed, 01 Sep 2021 04:13:37 GMT

If you're still getting this segfault, the developers have asked the following questions:

1) Which scale did you run ? "-n" and "-ppn".

2) We will be glad to get full log of running application with debug knobs:

export I_MPI_DEBUG=1000

export FI_LOG_LEVEL=debug

export I_MPI_HYDRA_DEBUG=1

Re:SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

Jennifer_D_Intel — Tue, 21 Sep 2021 20:58:45 GMT

I have an update on a fix for this. Starting in the IMPI 2021.3 release there is a knob for this issue:

I_MPI_SPAWN_EXPERIMENTAL=1

Setting that should resolve the problem.

Re: SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

menno — Tue, 07 Dec 2021 10:25:34 GMT

Sorry for not posting back earlier, I was not notified of any replies.

I will try the fix I_MPI_SPAWN_EXPERIMENTAL as soon as I have the 2021.3 release; currently we only have 2020 Update2 as most recent.

In the meantime, can you confirm that this is also fixed in the Intel oneAPI HPC pack?

Re:SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

Jennifer_D_Intel — Tue, 14 Dec 2021 18:45:30 GMT

Yes, this fix is included in the oneAPI HPC toolkit.

Re: SEGFAULT on repeated MPI_Comm_connect|MPI_Comm_accept/MPI_Comm_disconnect

menno — Tue, 11 Jan 2022 10:27:19 GMT

I can confirm that simulations now run with the proposed environment variable set.

A segfault occurs at the end of the simulations though, see here