[mpiexec@asrv0de102.corpdir.zz] Launch arguments: /clusterhead/projects/magma/v6.0.0.2-25420/v6.0.0/LINUX64/impi/bin//hydra_bstrap_proxy --upstream-host asrv0de102.corpdir.zz --upstream-port 44722 --pgid 0 --launcher ssh --launcher-number 0 --base-path /clusterhead/projects/magma/v6.0.0.2-25420/v6.0.0/LINUX64/impi/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /clusterhead/projects/magma/v6.0.0.2-25420/v6.0.0/LINUX64/impi/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9 [mpiexec@asrv0de102.corpdir.zz] Launch arguments: /usr/bin/ssh -q -x asrv0de103 /clusterhead/projects/magma/v6.0.0.2-25420/v6.0.0/LINUX64/impi/bin//hydra_bstrap_proxy --upstream-host asrv0de102.corpdir.zz --upstream-port 44722 --pgid 0 --launcher ssh --launcher-number 0 --base-path /clusterhead/projects/magma/v6.0.0.2-25420/v6.0.0/LINUX64/impi/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 1 --node-id 1 --subtree-size 1 /clusterhead/projects/magma/v6.0.0.2-25420/v6.0.0/LINUX64/impi/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9 [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@asrv0de102.corpdir.zz] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=get_maxes [proxy:0:0@asrv0de102.corpdir.zz] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=get_appnum [proxy:0:0@asrv0de102.corpdir.zz] PMI response: cmd=appnum appnum=0 [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@asrv0de102.corpdir.zz] PMI response: cmd=my_kvsname kvsname=kvs_15259_0 MPI startup(): Run 'pmi_process_mapping' nodemap algorithm [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=get kvsname=kvs_15259_0 key=PMI_process_mapping [proxy:0:0@asrv0de102.corpdir.zz] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [0] MPI startup(): Intel(R) MPI Library, Version 2021.10 Build 20230619 (id: c2e19c2f3e) [0] MPI startup(): Copyright (C) 2003-2023 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=barrier_in [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:1@asrv0de103.corpdir.zz] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=get_maxes [proxy:0:1@asrv0de103.corpdir.zz] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=get_appnum [proxy:0:1@asrv0de103.corpdir.zz] PMI response: cmd=appnum appnum=0 [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=get_my_kvsname [proxy:0:1@asrv0de103.corpdir.zz] PMI response: cmd=my_kvsname kvsname=kvs_15259_0 [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=get kvsname=kvs_15259_0 key=PMI_process_mapping [proxy:0:1@asrv0de103.corpdir.zz] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@asrv0de102.corpdir.zz] PMI response: cmd=barrier_out [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=barrier_in [proxy:0:1@asrv0de103.corpdir.zz] PMI response: cmd=barrier_out [0] MPI startup(): libfabric loaded: libfabric.so.1 [0] MPI startup(): libfabric version: 1.18.0-impi libfabric:15264:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_CUDA not supported libfabric:15264:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_ROCR not supported libfabric:27535:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_CUDA not supported libfabric:27535:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_ROCR not supported libfabric:15264:1697807008::core:core:ze_hmem_dl_init():497 Failed to dlopen libze_loader.so libfabric:27535:1697807008::core:core:ze_hmem_dl_init():497 Failed to dlopen libze_loader.so libfabric:27535:1697807008::core:core:ofi_hmem_init():421 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:15264:1697807008::core:core:ofi_hmem_init():421 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:27535:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_NEURON not supported libfabric:27535:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_SYNAPSEAI not supported libfabric:15264:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_NEURON not supported libfabric:15264:1697807008::core:core:ofi_hmem_init():416 Hmem iface FI_HMEM_SYNAPSEAI not supported libfabric:15264:1697807008::core:mr:ofi_default_cache_size():79 default cache size=7507620522 libfabric:27535:1697807008::core:mr:ofi_default_cache_size():79 default cache size=7507620522 libfabric:27535:1697807008::core:core:ofi_register_provider():476 registering provider: ofi_hook_noop (118.0) libfabric:27535:1697807008::core:core:ofi_register_provider():476 registering provider: off_coll (118.0) libfabric:27535:1697807008::core:core:fi_getinfo_():1338 Can't find provider with the highest priority libfabric:15264:1697807008::core:core:ofi_register_provider():476 registering provider: ofi_hook_noop (118.0) libfabric:15264:1697807008::core:core:ofi_register_provider():476 registering provider: off_coll (118.0) libfabric:15264:1697807008::core:core:fi_getinfo_():1338 Can't find provider with the highest priority Abort(2139535) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(176)........: MPID_Init(1548)..............: MPIDI_OFI_mpi_init_hook(1592): open_fabric(2650)............: find_provider(2794)..........: OFI fi_getinfo() failed (ofi_init.c:2794:find_provider:No data available) [proxy:0:0@asrv0de102.corpdir.zz] pmi cmd from fd 6: cmd=abort exitcode=2139535 [proxy:0:1@asrv0de103.corpdir.zz] pmi cmd from fd 4: cmd=abort exitcode=2139535 Abort(2139535) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(176)........: MPID_Init(1548)..............: MPIDI_OFI_mpi_init_hook(1592): open_fabric(2650)............: find_provider(2794)..........: OFI fi_getinfo() failed (ofi_init.c:2794:find_provider:No data available) [mpiexec@asrv0de102.corpdir.zz] Exit codes: [asrv0de102:0] 547720960 [asrv0de103:0] 547720960