InteloneAPI MPI 2021.2.0 behavior on Linux and Windows differ

[Frank_R_1]

Hi,

We are using Intel oneAPI MPI 2021.2.0 on Linux RHEL7 and Windows 10 in our product.

Calling, on Windows

mpiexec.exe -np x -delegate -localroot program args

and on Linux

mpiexec -np #x program args

we have the following parent child process structure:

mpiexec calls hydra_bstrap_proxy, which calls hydra_pmi_proxy and then ends itself.

hydra_pmi_proxy has then #x childs named program.

The process tree looks then like this (same on Windows and Linux)

mpiexec

hydra_pmi_proxy

|->program

|...

|->program

This means hydra_pmi_proxy is not a child of mpiexec like e.g. in older Intel MPI 2018.

Now to the different behavior:

If we kill the process mpiexec on Linux, hydra_pmi_proxy and its childs also gets ended.

If we kill the process mpiexec.exe on Windows (e.g. in process explorer), hydra_pmi_proxy.exe and its childs still live and run.

This is a big problem for our product. If customer needs to kill the running job, only mpiexec.exe gets killed but the hydra_pmi_proxy.exe and its childs still live and run, which is not the desired goal and leads to problems if our product gets restarted.

Why is the behavior with hydra_bstrap_proxy introduced? What is the benefit to not have a clear parent child process tree?

On Intel MPI 2018 we could kill mpiexec.exe and all childs in the process tree gets killed too.

Now behavior is odd, and we can't imagine why this was introduced

What guarantees that the kill of mpiexec.exe also ends the independed hydra_pmi_proxy.exe and its childs?

Best regards

Frank

[SantoshY_Intel]

Hi,

Thanks for reaching out to us.

We are working on your issue and we will get back to you soon.

Thanks & Regards,

Santosh

[Mark_L_Intel]

Hello Frank,

Sounds like a potential bug -- it should be a proper clean-up. As far as a version is concerned, IMPI 2019 version is based on new code base so you might see different behavior comparing to 2018 version.

[Mark_L_Intel · Accepted solution]

This behavior is common for Windows applications. For example, if you launch an application inside PowerShell, killing this PowerShell process may not kill your application which will continue to execute.  However, our development team decided to work out solution for gracefully aborting of MPI application on Windows. Thank you for contributing in improvement of Intel MPI! 

[Sachin_m_]

Hello Mark, 

Is the solution implemented in a fix for Intel MPI 2021.2? What version is this expected to be available in?

Thank you,

-Sachin

[Frank_Illenseer]

Hello Mark,

I would also be interested in knowing, if this is supposed to be resolved in a newer version and if "yes", then please specify on the exact version.

At least the 2021 update 2 version that we use seems to still have this behaviour!

Thanks and best regards,

Frank