HYD_connect_to_service (bstrap\service\service_launch.c:76): assert (!closed) failed

[Randykos]

Windows Server 2019  Version 1809 (OS Build 17763.4499)

CPU: dual socket Intel Xeon Gold 6142

Intel MPI 2018u3 and 2021.9 installed to default locations.

C:\>SET PATH=C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\bin;%PATH%
C:\>SET LIB=C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\lib;%LIB%
C:\>SET INCLUDE=C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\include;%INCLUDE%

C:\>mpiexec -V
Intel(R) MPI Library for Windows* OS, Version 2021.9 Build 20230308
Copyright 2003-2023, Intel Corporation.

C:\>hydra_service -status
[] HYD_connect_to_service (bstrap\service\service_launch.c:76): assert (!closed) failed
no hydra service running on machinename01

C:\>mpiexec -n 2 -host localhost "c:\temp\intelmpi_test.exe"
Intel MPI
Hello world! I'm rank 0 of 1 running on machinename.win.domain.com
Intel MPI
Hello world! I'm rank 0 of 1 running on machinename01.win.domain.com

Intel MPI 2018u3 and 2021.9 hydra services running on the computer.

"netstat -abo" shows Intel MPI 2018u3 running on port 8679 and Intel MPI 2021.9 running on port 8680

TCP 0.0.0.0:8679 machinename:0 LISTENING 4184
[hydra_service.exe]
TCP 0.0.0.0:8680 machinename:0 LISTENING 4104
[hydra_service.exe]

C:\>sc qc impi_hydra
[SC] QueryServiceConfig SUCCESS

SERVICE_NAME: impi_hydra
TYPE : 10 WIN32_OWN_PROCESS
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 0 IGNORE
BINARY_PATH_NAME : "C:\Program Files (x86)\IntelSWTools\mpi\2018.3.210\intel64\bin\hydra_service.exe"
LOAD_ORDER_GROUP :
TAG : 0
DISPLAY_NAME : Intel(R) MPI Library Hydra Process Manager
DEPENDENCIES :
SERVICE_START_NAME : LocalSystem

C:\>sc qc impi_hydra_2021_9_0
[SC] QueryServiceConfig SUCCESS

SERVICE_NAME: impi_hydra_2021_9_0
TYPE : 10 WIN32_OWN_PROCESS
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 0 IGNORE
BINARY_PATH_NAME : "C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\bin\hydra_service.exe" -service
LOAD_ORDER_GROUP :
TAG : 0
DISPLAY_NAME : Intel(R) MPI Library Hydra Process Manager 2021.9
DEPENDENCIES :
SERVICE_START_NAME : LocalSystem

C:\>sc query impi_hydra
SERVICE_NAME: impi_hydra
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

C:\>sc query impi_hydra_2021_9_0
SERVICE_NAME: impi_hydra_2021_9_0
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0

[Randykos]

Another simple "notepad" test on local computer works - when specifying localhost or the actual hostname.

However, when submitting to a remote computer, it fails with the same error message from "hydra_service -status"

note: if same notepad test run locally on the remote computer, it works as expected.


C:\>mpiexec.exe -n 1 -hosts RemoteComputer "C:\Windows\System32\Notepad.exe"
[mpiexec@LocalComputer] HYD_connect_to_service (bstrap\service\service_launch.c:76): assert (!closed) failed
[mpiexec@LocalComputer] HYDI_bstrap_service_launch (bstrap\service\service_launch.c:319): unable to connect to hydra service (RemoteComputer:8680)
[mpiexec@LocalComputer] remote_launch (bstrap\src\intel\i_hydra_bstrap.c:611): error launching bstrap proxy
[mpiexec@LocalComputer] single_launch (bstrap\src\intel\i_hydra_bstrap.c:669): remote launch error
[mpiexec@LocalComputer] launch_bstrap_proxies (bstrap\src\intel\i_hydra_bstrap.c:853): single launch error
[mpiexec@LocalComputer] HYD_bstrap_setup (bstrap\src\intel\i_hydra_bstrap.c:1047): unable to launch bstrap proxy
[mpiexec@LocalComputer] HYD_print_bstrap_setup_error_message (intel\i_mpiexec.c:1027): error setting up the bootstrap proxies
[mpiexec@LocalComputer] Possible reasons:
[mpiexec@LocalComputer] 1. Host is unavailable. Please check that all hosts are available.
[mpiexec@LocalComputer] 2. Cannot launch hydra_bstrap_proxy.exe or it crashed on one of the hosts. Make sure hydra_bstrap_proxy.exe is available on all hosts and it has right permissions.
[mpiexec@LocalComputer] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
[mpiexec@LocalComputer] 4. service bootstrap cannot launch processes on remote host. You may try using -bootstrap option to select alternative launcher.

However, TNC shows the port is available on the RemoteComputer.
C:\>powershell -Command "TNC cdc02winlm01 -Port 8680"

ComputerName : RemoteComputer
RemoteAddress : 10.2.3.102
RemotePort : 8680
InterfaceAlias : Private
SourceAddress : 10.2.3.102
TcpTestSucceeded : True

[AishwaryaCV_Intel]

Hi,

Thank you for posting in intel community.

The user authentication process has changed, hydra_service is only necessary for domain-based authentication, refer to the manual for details.

https://www.intel.com/content/www/us/en/developer/articles/release-notes/mpi-library-release-notes-windows.html

Also, you may find in our prerequisite steps that there is no need to install the service unless domain-based authentication is used

https://www.intel.com/content/www/us/en/docs/mpi-library/developer-guide-windows/2021-6/prerequisite-steps.html

Basically, we may recommend just installing and running IMPI without any extra prerequisite steps.

>>>>note: if same notepad test run locally on the remote computer, it works as expected.

Could you please provide the following details of remote computer:

1. CPU ,OS and hardware 
2. Sample reproducer(Notepad.exe) and steps to reproduce at our end.

Thanks And Regards,

Aishwarya 

[Randykos]

Hi Aishwarya,
We are using Domain authentication.

AD authentication is working fine for Intel MPI 2018U3 on the Windows HPC cluster.

ref: Intel 2018u3 tests
"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2018.3.210\windows\mpi\intel64\bin\mpivars.bat"

Intel(R) MPI Library 2018 Update 3 for Windows* Target Build Environment for Intel(R) 64 applications
Copyright 2007-2018 Intel Corporation.

mpiexec -V
Intel(R) MPI Library for Windows* OS, Version 2018 Update 3 Build 20180411
Copyright 2003-2018 Intel Corporation.

hydra_service -status
hydra service running on <hostname>.win.<domain>.com

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2018.3.210\windows\mpi\intel64\bin>mpiexec -n 2 -host <hostname1> "c:\temp\intelmpi_test.exe" : -n 2 -host <hostname2> "C:\temp\intelmpi_test.exe"
Intel MPI
Hello world! I'm rank 0 of 4 running on <hostname1>.win.<domain>.com
Hello world! I'm rank 1 of 4 running on <hostname1>.win.<domain>.com
Hello world! I'm rank 2 of 4 running on <hostname2>.win.<domain>.com
Hello world! I'm rank 3 of 4 running on <hostname2>.win.<domain>.com

ref: Intel 2021.9 tests

"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\env\vars.bat"

setspn.exe -A impi_hydra/%COMPUTERNAME%:8680/impi_hydra %COMPUTERNAME%

hydra_service -register_spn

C:\WINDOWS\system32>hydra_service -register_spn
Service Principal Name registered with the domain controller.
Hydra is now capable of launching processes using passwordless delegation.
The system administrator must ensure the following:
1) This host is trusted for delegation in Active Directory
2) All users who will run jobs are trusted for delegation.
Domain administrators can enable these options for hosts and users
in Active Directory on the domain controller.

However, the failure persists

C:\WINDOWS\system32>hydra_service -status
[] HYD_connect_to_service (bstrap\service\service_launch.c:76): assert (!closed) failed

no hydra service running on <hostname>

C:\>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\env\vars.bat"

C:\>mpiexec -n 2 -host <hostname1> "c:\temp\intelmpi_test.exe" : -n 2 -host <hostname2> "C:\temp\intelmpi_test.exe"
[mpiexec@<hostname1>] HYD_connect_to_service (bstrap\service\service_launch.c:76): assert (!closed) failed
[mpiexec@<hostname1>] HYDI_bstrap_service_launch (bstrap\service\service_launch.c:319): unable to connect to hydra service (<hostname2>:8680)
[mpiexec@<hostname1>] remote_launch (bstrap\src\intel\i_hydra_bstrap.c:611): error launching bstrap proxy
[mpiexec@<hostname1>] single_launch (bstrap\src\intel\i_hydra_bstrap.c:669): remote launch error
[mpiexec@<hostname1>] launch_bstrap_proxies (bstrap\src\intel\i_hydra_bstrap.c:853): single launch error
[mpiexec@<hostname1>] HYD_bstrap_setup (bstrap\src\intel\i_hydra_bstrap.c:1047): unable to launch bstrap proxy
[mpiexec@<hostname1>] HYD_print_bstrap_setup_error_message (intel\i_mpiexec.c:1027): error setting up the bootstrap proxies
[mpiexec@<hostname1>] Possible reasons:
[mpiexec@<hostname1>] 1. Host is unavailable. Please check that all hosts are available.
[mpiexec@<hostname1>] 2. Cannot launch hydra_bstrap_proxy.exe or it crashed on one of the hosts. Make sure hydra_bstrap_proxy.exe is available on all hosts and it has right permissions.
[mpiexec@<hostname1>] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
[mpiexec@<hostname1>] 4. service bootstrap cannot launch processes on remote host. You may try using -bootstrap option to select alternative launcher.

We are running 

Windows Server 2019  Version 1809 (OS Build 17763.4499)

Windows HPC Pack 2019 (build 6.1.7531.0)

Dual socket Intel Xeon Gold 6142 CPUs
Mellanox ConnectX-5 Adapter
Intel Ethernet X710 for 10GbE SFP+

The relevant environment variables have been set:
I_MPI_AUTH_METHOD=delegate
I_MPI_ONEAPI_ROOT=C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0
I_MPI_ROOT=C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\env\..
I_MPI_SMPD_VERSION_CHECK=0
I_MPI_TCP_NETMASK=10.2.1.0/20
ONEAPI_ROOT=C:\Program Files (x86)\Intel\oneAPI\

Thank you for any insight!

[Randykos]

Hi Aishwarya,

Did you see my response regarding Domain authentication?
It is working for IMPI 2018U3
However, it is not for IMPI 2021.9

thanks
Randy

[AishwaryaCV_Intel]

Hi,

Yes, Thanks for providing the information .

We are working on the issue , we will get back to you soon.

Thanks And Regards,

Aishwarya

[Randykos]

Hi Aishwarya,

We are still experiencing the issue, would you happen to have any suggestions.

thanks

Randy

[AishwaryaCV_Intel]

Hi,

Could you please try to check and run on Intel MPI version of 2021.8 and let us know if your facing the same issue?

Thanks And Regards,

Aishwarya

[Randykos]

Hi Aishwarya,

I am failing to locate the 2021.8 download.

However, I had an offline version of 2021.6 and it appears to work as expected.

I manually stopped the 2021.9 service and uninstalled it:
C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.9.0\bin\hydra_service.exe" -uninstall
Intel(R) MPI Library Hydra Process Manager 2021.9 removed.

Then, I installed the 2021.6 service:

C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin\hydra_service.exe" -install
Intel(R) MPI Library Hydra Process Manager 2021.6 installed.

C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin\mpiexec.exe" -n 2 -ppn 1 -hosts <hostname1>,<hostname2>hostname

resulting in:
<hostname2>
<hostname1>

verbose output:
C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin\mpiexec.exe" -v -n 2 -ppn 1 -hosts <hostname1>,<hostname2> hostname
[mpiexec@<hostname1>] Launch arguments: C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin\hydra_bstrap_proxy.exe --upstream-host <hostname1> --upstream-port 64300 --pgid 0 --launcher service --launcher-number 0 --base-path C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --service_port 0 --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 548 C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin\hydra_pmi_proxy.exe --usize -1 --auto-cleanup 1 --abort-signal 9
<hostname2>
<hostname1>


If you still want me to test with 2021.8, please share the download.

thanks

Randy

[Randykos]

Hi Aishwarya,

I was able to obtain the 2021.8 offline installer from a colleague.

C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.6.0\bin\hydra_service.exe" -uninstall
Stopping Intel(R) MPI Library Hydra Process Manager 2021.6.
Intel(R) MPI Library Hydra Process Manager 2021.6 stopped.
Intel(R) MPI Library Hydra Process Manager 2021.6 removed.

C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\hydra_service.exe" -install
Intel(R) MPI Library Hydra Process Manager 2021.8 installed and started.

Local machine test:

C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\mpiexec.exe" -v -n 2 -ppn 1 -hosts <hostname1> hostname
[mpiexec@<hostname1>] Launch arguments: C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\hydra_bstrap_proxy.exe --upstream-host <hostname1> --upstream-port 64872 --pgid 0 --launcher service --launcher-number 0 --base-path C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --service_port 0 --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 528 C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\hydra_pmi_proxy.exe --usize -1 --auto-cleanup 1 --abort-signal 9
<hostname1>
<hostname1>

Local and remote machine test:

C:\WINDOWS\system32>"C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\mpiexec.exe" -v -n 2 -ppn 1 -hosts <hostname1>,<hostname2> hostname
[mpiexec@<hostname1>] Launch arguments: C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\hydra_bstrap_proxy.exe --upstream-host <hostname1> --upstream-port 64862 --pgid 0 --launcher service --launcher-number 0 --base-path C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --service_port 0 --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 580 C:\Program Files (x86)\Intel\oneAPI\mpi\2021.8.0\bin\hydra_pmi_proxy.exe --usize -1 --auto-cleanup 1 --abort-signal 9
[mpiexec@<hostname1>] HYD_sspi_init_context (..\windows\src\hydra_sspi.c:114): InitializeSecurityContext failed(-2146893044)
[mpiexec@<hostname1>] HYD_send_sspi_credentials (bstrap\service\service_launch.c:205): unable to create sspi context
[mpiexec@<hostname1>] HYD_send_credentials_to_service (bstrap\service\service_launch.c:305): unable to send sspi credentials
[mpiexec@<hostname1>] HYDI_bstrap_service_launch (bstrap\service\service_launch.c:427): unable to send credentials to service
[mpiexec@<hostname1>] remote_launch (bstrap\src\intel\i_hydra_bstrap.c:607): error launching bstrap proxy
[mpiexec@<hostname1>] single_launch (bstrap\src\intel\i_hydra_bstrap.c:665): remote launch error
[mpiexec@<hostname1>] launch_bstrap_proxies (bstrap\src\intel\i_hydra_bstrap.c:849): single launch error
[mpiexec@<hostname1>] HYD_bstrap_setup (bstrap\src\intel\i_hydra_bstrap.c:1043): unable to launch bstrap proxy
[mpiexec@<hostname1>] HYD_print_bstrap_setup_error_message (intel\i_mpiexec.c:1027): error setting up the bootstrap proxies
[mpiexec@<hostname1>] Possible reasons:
[mpiexec@<hostname1>] 1. Host is unavailable. Please check that all hosts are available.
[mpiexec@<hostname1>] 2. Cannot launch hydra_bstrap_proxy.exe or it crashed on one of the hosts. Make sure hydra_bstrap_proxy.exe is available on all hosts and it has right permissions.
[mpiexec@<hostname1>] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
[mpiexec@<hostname1>] 4. service bootstrap cannot launch processes on remote host. You may try using -bootstrap option to select alternative launcher.

Port is available

M:\>powershell -Command "TNC <hostname1> -Port 8680

ComputerName : <hostname1>
RemoteAddress : 10.233.22.101
RemotePort : 8680
InterfaceAlias : Private
SourceAddress : 10.233.22.127
TcpTestSucceeded : True

M:\>powershell -Command "TNC <hostname2> -Port 8680

ComputerName : <hostname2>
RemoteAddress : 10.233.22.108
RemotePort : 8680
InterfaceAlias : Private
SourceAddress : 10.233.22.127
TcpTestSucceeded : True

[Randykos]

Hi Aishwarya,

I installed Intel MPI 2021.10, this also fails the test.

Why would Intel 2018U3 and 2021.6 show functionallity but the later releases fail.

thanks

Randy

[Randykos]

Hi Aishwarya,

Do you have any suggestions?

thanks

Randy

[Randykos]

Hi Aishwarya,
We are still facing the issue, do you have any suggestions?
thanks
Randy

[Randykos]

Hi Aishwarya,
We are still stuck on this issue - please provide guidance.

thanks

Randy