^C J:\test\x64\Debug>set I_MPI_DEBUG=20 J:\test\x64\Debug>mpiexec -n 8 -ppn 4 -hosts mepl88 test.exe [0] MPI startup(): Intel(R) MPI Library, Version 2021.5 Build 20211102 [0] MPI startup(): Copyright (C) 2003-2021 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): shm segment size (298 MB per rank) * (8 local ranks) = 2390 MB total [0] MPI startup(): libfabric version: 1.13.2-impi libfabric:12436:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:12436:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:12436:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:12436:netdir:core:ofi_nd_startup():602 ofi_nd_startup: starting initialization libfabric:12436:core:core:ofi_register_provider():474 registering provider: netdir (113.20) libfabric:12436:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:12436:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:12436:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:12436:core:core:ofi_register_provider():474 registering provider: ofi_hook_perf (113.20) libfabric:12436:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:fi_getinfo():1123 Can't find provider with the highest priority libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priorit[0] MPI startup(): libfabric provider: tcp;ofi_rxm [0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1 [0] MPI startup(): addrnamelen: zu [0] MPI startup(): File "/tuning_skx_shm-ofi_tcp-ofi-rxm.dat" not found [0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat" [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for ch4 level [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for net level [0] MPI startup(): File "" not found [0] MPI startup(): Unable to read tuning file for shm level [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 12436 mepl88 0 [0] MPI startup(): 1 12584 mepl88 1 [0] MPI startup(): 2 12564 mepl88 2 [0] MPI startup(): 3 12604 mepl88 3 [0] MPI startup(): 4 12728 mepl88 4 [0] MPI startup(): 5 12712 mepl88 5 [0] MPI startup(): 6 12744 mepl88 6 [0] MPI startup(): 7 12788 mepl88 7 [0] MPI startup(): I_MPI_ONEAPI_ROOT=J:\oneAPI\mpi\2021.5.0 [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_DEBUG=20 [0] MPI startup(): threading: mode: direct [0] MPI startup(): threading: vcis: 1 [0] MPI startup(): threading: app_threads: 1 [0] MPI startup(): threading: runtime: generic [0] MPI startup(): threading: is_threaded: 0 [0] MPI startup(): threading: async_progress: 0 [0] MPI startup(): threading: num_pools: 64 [0] MPI startup(): threading: lock_level: global [0] MPI startup(): threading: enable_sep: 0 [0] MPI startup(): threading: direct_recv: 1 [0] MPI startup(): threading: zero_op_flags: 0 [0] MPI startup(): threading: num_am_buffers: 8 [0] MPI startup(): threading: library is built with per-vci thread granularity Hello world: rank 0 of 8 running on mepl88 Hello world: rank 1 of 8 running on mepl88 Hello world: rank 2 of 8 running on mepl88 Hello world: rank 3 of 8 running on mepl88 Hello world: rank 4 of 8 running on mepl88 Hello world: rank 5 of 8 running on mepl88 Hello world: rank 6 of 8 running on mepl88 Hello world: rank 7 of 8 running on mepl88 y tcp, must_use_util_prov = 1 libfabric:12436:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1201 Start regular provider search because provider with the highest priority tcp can not be initialized libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_fabric():1423 Opened fabric: 192.168.0.0/24 libfabric:12436:core:core:fi_fabric():1423 Opened fabric: 192.168.0.0/24 libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12436:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12436:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets J:\test\x64\Debug>mpiexec -n 8 -ppn 4 -hosts mepl88 test.exe > 1.out libfabric:12856:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:12856:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:12856:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:12856:netdir:core:ofi_nd_startup():602 ofi_nd_startup: starting initialization libfabric:12856:core:core:ofi_register_provider():474 registering provider: netdir (113.20) libfabric:12856:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:12856:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:12856:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:12856:core:core:ofi_register_provider():474 registering provider: ofi_hook_perf (113.20) libfabric:12856:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:fi_getinfo():1123 Can't find provider with the highest priority libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1201 Start regular provider search because provider with the highest priority tcp can not be initialized libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_fabric():1423 Opened fabric: 192.168.0.0/24 libfabric:12856:core:core:fi_fabric():1423 Opened fabric: 192.168.0.0/24 libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:12856:core:core:fi_getinfo():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir libfabric:12856:core:core:fi_getinfo():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets J:\test\x64\Debug>mpiexec -n 8 -ppn 4 -hosts meetingroom test.exe > node1.txt [mpiexec@mepl88] HYD_sock_connect (..\windows\src\hydra_sock.c:240): Retrying connection, retry_count=1, retries=0 [mpiexec@mepl88] HYD_connect_to_service (bstrap\service\service_launch.c:85): assert (!closed) failed [mpiexec@mepl88] HYDI_bstrap_service_launch (bstrap\service\service_launch.c:419): unable to connect to hydra service (meetingroom:8680) [mpiexec@mepl88] remote_launch (bstrap\src\intel\i_hydra_bstrap.c:607): error launching bstrap proxy [mpiexec@mepl88] single_launch (bstrap\src\intel\i_hydra_bstrap.c:665): remote launch error [mpiexec@mepl88] launch_bstrap_proxies (bstrap\src\intel\i_hydra_bstrap.c:849): single launch error [mpiexec@mepl88] HYD_bstrap_setup (bstrap\src\intel\i_hydra_bstrap.c:1043): unable to launch bstrap proxy [mpiexec@mepl88] HYD_print_bstrap_setup_error_message (intel\i_mpiexec.c:1031): error setting up the bootstrap proxies [mpiexec@mepl88] Possible reasons: [mpiexec@mepl88] 1. Host is unavailable. Please check that all hosts are available. [mpiexec@mepl88] 2. Cannot launch hydra_bstrap_proxy.exe or it crashed on one of the hosts. Make sure hydra_bstrap_proxy.exe is available on all hosts and it has right permissions. [mpiexec@mepl88] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable. [mpiexec@mepl88] 4. service bootstrap cannot launch processes on remote host. You may try using -bootstrap option to select alternative launcher.