Loading mpi version 2021.4.0 Intel(R) MPI Library for Linux* OS, Version 2021.4 Build 20210831 (id: 758087adf) Copyright 2003-2021, Intel Corporation. [mpiexec@node02] Launch arguments: /opt/slurm/current/bin/srun -N 1 -n 1 --ntasks-per-node 1 --nodelist node02 --input none /opt/intel/oneapi/mpi/2021.4.0/bin//hydra_bstrap_proxy --upstream-host node02.cm.cluster --upstream-port 34395 --pgid 0 --launcher slurm --launcher-number 1 --base-path /opt/intel/oneapi/mpi/2021.4.0/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 0 --debug /opt/intel/oneapi/mpi/2021.4.0/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9 IPL WARN> ipl_create_domains: Not enough cpus for the specified I_MPI_PIN_ORDER, switch to I_MPI_PIN_ORDER=compact [proxy:0:0@node02] pmi cmd from fd 4: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 5: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 6: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 9: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 13: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 33: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [0] MPI startup(): Intel(R) MPI Library, Version 2021.4 Build 20210831 (id: 758087adf) [0] MPI startup(): Copyright (C) 2003-2021 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [proxy:0:0@node02] pmi cmd from fd 4: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 5: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 6: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 13: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 33: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 4: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 5: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 6: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 9: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 13: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 17: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 4: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 5: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 6: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 9: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 13: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 33: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 4: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 5: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 6: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 9: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 13: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 17: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 9: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 33: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 17: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 33: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 17: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 17: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 21: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 21: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 21: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 21: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 21: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 25: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [0] MPI startup(): shm segment size (494 MB per rank) * (10 local ranks) = 4942 MB total [proxy:0:0@node02] pmi cmd from fd 25: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 4: cmd=put kvsname=kvs_678785_0 key=-bcast-1-0 value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@node02] pmi cmd from fd 5: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 6: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 9: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 13: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 17: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 21: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 25: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 29: cmd=init pmi_version=1 pmi_subversion=1 [proxy:0:0@node02] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@node02] pmi cmd from fd 33: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 4: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 25: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 29: cmd=get_maxes [proxy:0:0@node02] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096 [proxy:0:0@node02] pmi cmd from fd 25: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 29: cmd=get_appnum [proxy:0:0@node02] PMI response: cmd=appnum appnum=0 [proxy:0:0@node02] pmi cmd from fd 29: cmd=get_my_kvsname [proxy:0:0@node02] PMI response: cmd=my_kvsname kvsname=kvs_678785_0 [proxy:0:0@node02] pmi cmd from fd 25: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 29: cmd=get kvsname=kvs_678785_0 key=PMI_process_mapping [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,10)) [proxy:0:0@node02] pmi cmd from fd 29: cmd=barrier_in [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] pmi cmd from fd 13: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 17: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 21: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 33: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 5: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 6: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 9: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 25: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [proxy:0:0@node02] pmi cmd from fd 29: cmd=get kvsname=kvs_678785_0 key=-bcast-1-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F49786D557149 [0] MPI startup(): NUMA nodes mask 0xf [1] MPI startup(): NUMA nodes mask 0xf [2] MPI startup(): NUMA nodes mask 0xf [3] MPI startup(): NUMA nodes mask 0xf [4] MPI startup(): NUMA nodes mask 0xf [5] MPI startup(): NUMA nodes mask 0xf [6] MPI startup(): NUMA nodes mask 0xf [7] MPI startup(): NUMA nodes mask 0xf [8] MPI startup(): NUMA nodes mask 0xf [9] MPI startup(): NUMA nodes mask 0xf [0] MPI startup(): libfabric version: 1.13.0-impi libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678802:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678803:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678804:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678805:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678806:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678807:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678808:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678809:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678810:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678811:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678802:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678803:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678804:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678805:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678806:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678807:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678808:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678809:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678810:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678811:core:mr:ofi_default_cache_size():76 default cache size=4222470720 libfabric:678802:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678802:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678802:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678802:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678804:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678804:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678804:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678804:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678805:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678805:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678805:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678805:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678807:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678807:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678807:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678807:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678808:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678808:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678808:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678808:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678809:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678809:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678809:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678809:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678811:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678811:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678811:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678811:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678803:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678803:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678803:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678803:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678806:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678806:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678806:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678806:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678810:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:678810:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:678810:verbs:fabric:verbs_devs_print():897 10.10.54.17 libfabric:678810:verbs:fabric:verbs_devs_print():897 fe80::1e34:da03:54:bf60 libfabric:678811:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678807:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678810:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678808:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678809:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678805:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678806:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678803:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678804:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678802:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678807:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678806:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678811:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678810:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678809:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678808:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678802:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678803:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678804:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678805:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678807:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678807:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678811:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678811:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678806:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678806:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678809:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678809:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678808:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678808:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678806:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678807:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678808:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678809:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678811:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678810:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678810:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678810:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678802:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678802:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678803:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678803:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678804:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678804:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678802:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678803:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678804:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678805:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:678805:core:core:ofi_register_provider():474 registering provider: verbs (113.0) libfabric:678805:core:core:ofi_register_provider():474 registering provider: tcp (113.0) libfabric:678802:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678803:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678804:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678805:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678806:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678807:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678808:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678809:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678810:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678811:core:core:ofi_register_provider():474 registering provider: sockets (113.0) libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678802:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678802:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678803:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678803:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678804:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678804:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678805:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678805:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678806:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678806:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678807:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678807:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678808:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678808:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678809:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678809:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678810:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678810:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:678811:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:678811:core:core:ofi_register_provider():474 registering provider: shm (113.0) libfabric:678802:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678803:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678804:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678805:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678806:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678807:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678808:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678809:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678810:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678811:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.0) libfabric:678802:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678803:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678804:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678805:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678806:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678807:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678808:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678809:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678810:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678811:core:core:ofi_register_provider():474 registering provider: psm2 (113.0) libfabric:678802:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678802:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678803:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678803:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678804:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678804:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678805:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678805:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678806:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678806:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678807:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678807:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678808:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678808:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678809:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678809:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678810:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678810:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678811:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:678811:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:678802:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678803:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678804:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678805:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678806:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678807:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678808:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678809:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678810:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678811:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:678802:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678802:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678802:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678803:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678803:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678803:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678804:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678804:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678804:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678805:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678805:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678805:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678806:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678806:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678806:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678807:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678807:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678807:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678808:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678808:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678808:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678809:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678809:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678809:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678810:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678810:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678810:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678811:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.0) libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():254 RMA only instance included libfabric:678811:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678811:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678802:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:psm3:core:psmx3_getinfo():502 libfabric:678802:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678802:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678802:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:psm3:core:psmx3_getinfo():502 libfabric:678803:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678803:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678803:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:psm3:core:psmx3_getinfo():502 libfabric:678804:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678804:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678804:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:psm3:core:psmx3_getinfo():502 libfabric:678805:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678805:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678805:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:psm3:core:psmx3_getinfo():502 libfabric:678806:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678806:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678806:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:psm3:core:psmx3_getinfo():502 libfabric:678807:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678807:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678807:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:psm3:core:psmx3_getinfo():502 libfabric:678808:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678808:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678808:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:psm3:core:psmx3_getinfo():502 libfabric:678809:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678809:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678809:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:psm3:core:psmx3_getinfo():502 libfabric:678810:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678810:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678810:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:psm3:core:psmx3_getinfo():502 libfabric:678811:psm3:core:psmx3_init_prov_info():413 RMA only instance included libfabric:678811:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678811:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678802:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678802:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678802:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678803:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678803:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678803:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678804:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678804:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678804:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678805:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678805:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678805:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678806:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678806:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678806:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678807:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678807:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678807:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678808:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678808:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678808:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678809:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678809:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678809:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678810:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678810:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678810:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678811:psm3:core:psmx3_init_lib():263 PSM3 header version = (3, 0) libfabric:678811:psm3:core:psmx3_init_lib():265 PSM3 library version = (3, 0) libfabric:678811:psm3:core:psmx3_init_lib():270 PSM3 multi-ep feature not available or disabled. libfabric:678805:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678805:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678803:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678803:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678806:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678806:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678807:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678807:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678809:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678809:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678810:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678810:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678808:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678808:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678811:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678811:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678802:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678802:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678804:psm3:core:psmx3_update_hfi_info():406 hfi1 units: total 1, active 1; hfi1 contexts: total 1024, free 1024 libfabric:678804:psm3:core:psmx3_update_hfi_info():418 Tx/Rx contexts: 1 in total, 1 available. libfabric:678803:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678805:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678806:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678802:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678804:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678807:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678810:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678811:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678808:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678809:psm3:core:psmx3_alter_prov_info():628 3 instances available, 2 with CQ data flag set libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678802:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678802:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678802:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678803:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678803:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678803:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678803:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678804:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678804:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678804:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678804:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678805:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678805:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678805:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678806:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678806:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678806:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678807:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678807:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678807:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678808:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678808:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678808:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678809:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678809:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678809:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678810:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678810:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678810:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678811:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678811:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678811:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678802:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678805:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678806:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678807:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678808:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678809:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678810:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678811:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678806:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678806:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678806:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678806:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678811:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678811:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678811:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678811:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678807:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678807:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678807:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678807:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678805:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678805:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678805:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678805:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678804:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678804:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678804:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678804:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678809:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678809:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678809:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678809:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678806:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678806:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678806:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678806:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678810:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678810:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678810:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678810:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678807:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678807:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678807:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678807:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678802:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678802:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678802:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678802:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678803:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678803:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678803:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678803:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678808:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678808:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678808:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:mlx:core:ofi_check_ep_attr():718 Shared rx context not supported libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678808:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678811:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678811:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678811:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678811:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678806:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678806:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678806:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678806:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678805:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678805:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678805:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678805:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678809:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678809:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678809:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678809:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678811:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678811:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678811:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678811:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678804:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678804:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678804:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678804:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678805:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678805:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678805:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678805:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678809:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678809:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678809:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678809:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678807:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678807:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678807:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678807:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678804:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678804:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678804:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678804:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678802:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678802:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678802:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678802:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678808:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678808:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678808:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678808:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678803:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678803:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678803:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678803:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678810:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678810:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678810:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678810:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678803:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678803:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678803:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678803:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678802:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678802:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678802:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678802:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678810:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678810:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678810:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678810:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678808:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678808:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678808:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678808:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678806:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678806:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:psm3:core:psmx3_getinfo():502 libfabric:678806:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678806:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678806:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678806:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678806:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678806:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678806:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678806:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678806:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678806:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678806:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678806:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0-dgram libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678811:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678811:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:psm3:core:psmx3_getinfo():502 libfabric:678811:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678811:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678811:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678811:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678811:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678811:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678811:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678811:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678811:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678811:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678811:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678811:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678806:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678806:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678806:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678806:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678806:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678805:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678805:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:psm3:core:psmx3_getinfo():502 libfabric:678805:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678805:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678805:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678805:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678805:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678805:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678805:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678805:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678805:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678805:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678805:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678805:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678807:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678807:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:psm3:core:psmx3_getinfo():502 libfabric:678807:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678807:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678807:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678807:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678807:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678807:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678807:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678807:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678807:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678807:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678807:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678807:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678804:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678804:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:psm3:core:psmx3_getinfo():502 libfabric:678804:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678804:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678804:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678804:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678804:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678804:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678804:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678804:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678804:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678804:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678804:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678804:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678809:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:psm3:core:psmx3_getinfo():502 libfabric:678809:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678809:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678809:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678809:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678809:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678809:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678809:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678809:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678809:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678809:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678809:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678809:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678806:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678806:psm2:core:psmx2_getinfo():523 libfabric:678806:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678806:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678806:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678806:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678806:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678806:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678806:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678806:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678806:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678811:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678811:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678811:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678811:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678811:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678809:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678809:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678809:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678809:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678809:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678805:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678805:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678805:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678805:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678805:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678804:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678804:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678804:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678804:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678804:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678811:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678811:psm2:core:psmx2_getinfo():523 libfabric:678811:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678811:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678811:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678811:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678811:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678811:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678811:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678811:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678811:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678807:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678807:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678807:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678807:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678807:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678803:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:psm3:core:psmx3_getinfo():502 libfabric:678803:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678803:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678803:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678803:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678803:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678803:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678803:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678803:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678803:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678803:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678803:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678803:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678807:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678807:psm2:core:psmx2_getinfo():523 libfabric:678807:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678807:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678807:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678807:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678807:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678807:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678807:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678807:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678807:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678807:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678804:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678804:psm2:core:psmx2_getinfo():523 libfabric:678804:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678804:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678804:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678804:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678804:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678804:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678804:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678804:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678804:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678804:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678809:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678809:psm2:core:psmx2_getinfo():523 libfabric:678809:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678809:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678809:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678809:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678809:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678809:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678809:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678809:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678809:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678809:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678806:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678806:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678806:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678806:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678806:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678806:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678806:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678810:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678810:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678803:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678803:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678803:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678803:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:psm3:core:psmx3_getinfo():502 libfabric:678810:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678810:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678810:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678810:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678810:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678810:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678810:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678810:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678810:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678810:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678810:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678810:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678803:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678811:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678811:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678811:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678811:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678811:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678811:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678811:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678811:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678806:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678806:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678806:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678805:psm2:core:psmx2_getinfo():523 libfabric:678805:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678805:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678805:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678805:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678805:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678805:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678805:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678805:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678805:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678805:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678808:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678806:mlx:core:mlx_fabric_open():172 libfabric:678806:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678806:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678806:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:psm3:core:psmx3_getinfo():502 libfabric:678808:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678808:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678808:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678808:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678808:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678808:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678808:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678808:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678808:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678808:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678808:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678808:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678807:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678807:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678807:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678807:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678807:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678807:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678807:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678807:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678809:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678809:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678809:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678809:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678809:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678809:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678809:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():268 TAG60 instance included libfabric:678802:psm2:core:psmx2_init_prov_info():281 TAG64 instance included libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:psm2:core:psmx2_init_lib():234 no hfi device is found. libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:psm3:core:psmx3_getinfo():502 libfabric:678802:psm3:core:psmx3_init_prov_info():427 TAG60 instance included libfabric:678802:psm3:core:psmx3_init_prov_info():440 TAG64 instance included libfabric:678802:psm3:core:psmx3_alter_prov_info():628 2 instances available, 1 with CQ data flag set libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority verbs, must_use_util_prov = 1 libfabric:678802:core:core:fi_getinfo_():1161 Since verbs can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678802:core:core:fi_getinfo_():1161 Since verbs can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678802:core:core:fi_getinfo_():1161 Since verbs can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1161 Since verbs can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:678802:core:core:fi_getinfo_():1161 Since verbs can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678802:core:core:fi_getinfo_():1161 Since verbs can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority tcp, must_use_util_prov = 1 libfabric:678802:core:core:fi_getinfo_():1161 Since tcp can be used, psm2 has been skipped. To use psm2, please, set FI_PROVIDER=psm2 libfabric:678802:core:core:fi_getinfo_():1161 Since tcp can be used, mlx has been skipped. To use mlx, please, set FI_PROVIDER=mlx libfabric:678802:core:core:fi_getinfo_():1161 Since tcp can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:678802:core:core:fi_getinfo_():1161 Since tcp can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:678811:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678811:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678811:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678811:mlx:core:mlx_fabric_open():172 libfabric:678811:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678811:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678811:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678805:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678805:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678805:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678805:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678805:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678805:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678805:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678805:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678805:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678805:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678805:mlx:core:mlx_fabric_open():172 libfabric:678805:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678805:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678805:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678807:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678807:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678807:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678807:mlx:core:mlx_fabric_open():172 libfabric:678807:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678807:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678807:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678802:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678802:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678802:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678802:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678802:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678803:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678803:psm2:core:psmx2_getinfo():523 libfabric:678803:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678803:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678803:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678803:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678803:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678803:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678803:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678803:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678803:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678803:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678804:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678804:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678804:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678804:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678804:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678804:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678804:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678809:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678809:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678809:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678809:mlx:core:mlx_fabric_open():172 libfabric:678809:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678809:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678809:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678803:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678803:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678803:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678803:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678803:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678803:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678803:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678803:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678803:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678803:mlx:core:mlx_fabric_open():172 libfabric:678803:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678803:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678803:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678802:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678802:psm2:core:psmx2_getinfo():523 libfabric:678802:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678802:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678802:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678802:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678802:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678802:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678802:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678802:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678802:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678802:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678810:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678810:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678810:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678810:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678810:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678804:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678804:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678804:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678804:mlx:core:mlx_fabric_open():172 libfabric:678804:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678804:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678804:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:fi_getinfo_():1161 Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:678808:core:core:fi_getinfo_():1161 Since tcp can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678808:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678808:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678808:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678808:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678802:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678802:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678802:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678802:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678802:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678802:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678808:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678808:psm2:core:psmx2_getinfo():523 libfabric:678808:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678808:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678808:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678808:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678808:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678808:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678808:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678808:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678808:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678808:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG [0] MPI startup(): libfabric provider: mlx [0] MPI startup(): detected mlx provider, set device name to "mlx" libfabric:678802:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678802:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678802:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678802:mlx:core:mlx_fabric_open():172 libfabric:678802:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678802:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678802:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678810:core:core:fi_getinfo_():1138 Found provider with the highest priority psm2, must_use_util_prov = 0 libfabric:678810:psm2:core:psmx2_getinfo():523 libfabric:678810:psm2:core:psmx2_init_prov_info():201 Unsupported endpoint type libfabric:678810:psm2:core:psmx2_init_prov_info():203 Supported: FI_EP_RDM libfabric:678810:psm2:core:psmx2_init_prov_info():205 Supported: FI_EP_DGRAM libfabric:678810:psm2:core:psmx2_init_prov_info():207 Requested: FI_EP_MSG libfabric:678810:core:core:fi_getinfo_():1201 Start regular provider search because provider with the highest priority psm2 can not be initialized libfabric:678810:mlx:core:mlx_getinfo():210 primary detected device: mlx5_0 libfabric:678810:mlx:core:mlx_getinfo():233 used inject size = 1024 libfabric:678810:mlx:core:mlx_getinfo():280 Loaded MLX version 1.11.1 libfabric:678810:mlx:core:mlx_getinfo():327 MLX: spawn support 0 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;psm3 layering libfabric:678810:core:core:ofi_layering_ok():1001 Need core provider, skipping ofi_rxm libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1557 adding fi_info for domain: mlx5_0 libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:fabric:vrb_get_matching_info():1534 hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_MSG libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678808:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678808:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678808:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678808:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678808:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678808:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678808:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678810:tcp:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:tcp:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:tcp:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;sockets layering libfabric:678810:core:core:ofi_layering_ok():1007 Skipping util;shm layering libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #1 mlx5_0 libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #2 mlx5_0 libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #3 mlx5_0-xrc libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #4 mlx5_0-xrc libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #5 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:verbs:fabric:vrb_get_matching_info():1512 checking domain: #6 mlx5_0-dgram libfabric:678810:verbs:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:verbs:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:verbs:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:tcp:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:tcp:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678810:tcp:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_DGRAM libfabric:678810:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:sockets:core:ofi_check_ep_type():658 unsupported endpoint type libfabric:678810:sockets:core:ofi_check_ep_type():659 Supported: FI_EP_MSG libfabric:678810:sockets:core:ofi_check_ep_type():659 Requested: FI_EP_RDM libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.140.54.17, iface name: eno8303, speed: 1000 libfabric:678810:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678810:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678810:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678810:mlx:core:mlx_fabric_open():172 libfabric:678810:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678810:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678810:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: 10.10.54.17, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::bacb:29ff:feaa:a9ae, iface name: eno8303, speed: 1000 libfabric:678808:sockets:core:ofi_get_list_of_addr():1720 Available addr: fe80::1e34:da03:54:bf60, iface name: ib0, speed: 200000 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1563 available addr: : fi_sockaddr_in://127.0.0.1:0 libfabric:678808:sockets:core:ofi_insert_loopback_addr():1578 available addr: : fi_sockaddr_in6://[::1]:0 libfabric:678808:sockets:core:util_getinfo_ifs():318 Chosen addr for using: 10.10.54.17, speed 200000 libfabric:678808:mlx:core:mlx_fabric_open():172 libfabric:678808:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:678808:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678808:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678806:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678806:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678807:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678807:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678811:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678811:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678803:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678803:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678805:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678805:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678809:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678809:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678804:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678804:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps [0] MPI startup(): max_ch4_vcis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 1 libfabric:678802:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678802:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678810:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678810:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678808:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:678808:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:678807:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678809:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678811:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... [proxy:0:0@node02] pmi cmd from fd 17: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 25: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 33: cmd=barrier_in [0] MPI startup(): addrnamelen: 1024 libfabric:678802:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678808:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678805:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... [proxy:0:0@node02] pmi cmd from fd 4: cmd=put kvsname=kvs_678785_0 key=bc-0 value=mpi#007793E1416715150B2000637577CC2B3200F8D74F00000000004F0300883800B15B5BCCE01B218300A302278AB00FA133EF48B8D0B00F213532DF2200007AD477CC2B33EF48B8D0B00F213532DF2200010031D15C7CE133EF48B8D0451D29353203270003E16801B30F77CCAB33EF48B8D0451D29353203270083E9680100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ [proxy:0:0@node02] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@node02] pmi cmd from fd 21: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 4: cmd=put kvsname=kvs_678785_0 key=-bcast-2-0 value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@node02] pmi cmd from fd 4: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 9: cmd=barrier_in libfabric:678803:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678804:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678806:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... libfabric:678810:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [126]... [proxy:0:0@node02] pmi cmd from fd 5: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 6: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 13: cmd=barrier_in [proxy:0:0@node02] pmi cmd from fd 29: cmd=barrier_in [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] PMI response: cmd=barrier_out [proxy:0:0@node02] pmi cmd from fd 9: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 5: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 6: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 13: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 17: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 21: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 25: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 29: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 [proxy:0:0@node02] pmi cmd from fd 33: cmd=get kvsname=kvs_678785_0 key=-bcast-2-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=2F6465762F73686D2F496E74656C5F4D50495F686767447865 libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f2000122dc0 [proxy:0:0@node02] pmi cmd from fd 33: cmd=get kvsname=kvs_678785_0 key=bc-0 [proxy:0:0@node02] PMI response: cmd=get_result rc=0 msg=success value=mpi#007793E1416715150B2000637577CC2B3200F8D74F00000000004F0300883800B15B5BCCE01B218300A302278AB00FA133EF48B8D0B00F213532DF2200007AD477CC2B33EF48B8D0B00F213532DF2200010031D15C7CE133EF48B8D0451D29353203270003E16801B30F77CCAB33EF48B8D0451D29353203270083E9680100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000$ libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f200004ce40 libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f200007c780 libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f2000197380 libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f20001e8540 libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f20000c9f40 libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f2000118580 libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f2000042ec0 libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f20001b3c40 libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=1) fi_addr=0x7f200010f480 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f200016e900 libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f2000098b00 libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f20000c8440 libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f20001e3040 libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f2000234200 libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f2000115c00 libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f2000164240 libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f200008eb80 libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f20001ff900 libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=9) fi_addr=0x7f200015b140 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f200016e900 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f2000098b00 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f20000c8440 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f20001e3040 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f2000234200 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f2000115c00 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f2000164240 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f200008eb80 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f20001ff900 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #1, offset=1024 (size=9) fi_addr=0x7f200015b140 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f20001e3040 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f2000234200 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f200008eb80 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f200015b140 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f200016e900 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f2000098b00 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f20000c8440 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f2000164240 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f20001ff900 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #2, offset=2048 (size=9) fi_addr=0x7f2000115c00 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f20001e3040 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f200008eb80 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f200015b140 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f2000234200 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f20001ff900 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f200016e900 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f20000c8440 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f2000115c00 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f2000164240 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f20001e3040 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f200015b140 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f2000234200 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f200008eb80 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f20001ff900 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f200016e900 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f2000115c00 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f2000164240 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f200008eb80 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f200015b140 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f2000234200 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f20001ff900 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f200016e900 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f2000164240 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f2000115c00 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f20001e3040 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f20001ff900 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f200015b140 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f200008eb80 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f200016e900 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f2000115c00 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f2000164240 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f20001e3040 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f20001ff900 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f200015b140 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f200016e900 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f2000164240 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f200008eb80 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f20001e3040 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f20000c8440 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678802:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f200016e900 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678810:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f20001ff900 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678805:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f20001e3040 libfabric:678805:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f2000234200 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f2000234200 libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f200008eb80 libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f200015b140 libfabric:678802:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f20000c8440 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f20000c8440 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f20000c8440 libfabric:678810:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678806:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f2000234200 libfabric:678806:mlx:core:mlx_av_insert():189 address inserted libfabric:678811:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678804:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f20000c8440 libfabric:678804:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678808:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f2000164240 libfabric:678808:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #3, offset=3072 (size=9) fi_addr=0x7f2000098b00 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #4, offset=4096 (size=9) fi_addr=0x7f2000098b00 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #5, offset=5120 (size=9) fi_addr=0x7f2000098b00 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #6, offset=6144 (size=9) fi_addr=0x7f2000098b00 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f2000098b00 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678803:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f2000098b00 libfabric:678803:mlx:core:mlx_av_insert():189 address inserted libfabric:678809:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #7, offset=7168 (size=9) fi_addr=0x7f2000115c00 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted libfabric:678807:mlx:core:mlx_av_insert():179 Try to insert address #8, offset=8192 (size=9) fi_addr=0x7f2000115c00 libfabric:678807:mlx:core:mlx_av_insert():189 address inserted [0] MPI startup(): Load tuning file: "/opt/intel/oneapi/mpi/2021.4.0/etc/tuning_icx_shm-ofi_mlx.dat" [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 678802 node02 {0,4,8,12,16,20} [0] MPI startup(): 1 678803 node02 {24,28,32,36,40,44} [0] MPI startup(): 2 678804 node02 {2,6,48,52,56,60} [0] MPI startup(): 3 678805 node02 {10,14,18,22,26,30} [0] MPI startup(): 4 678806 node02 {34,38,42,46,50,54} [0] MPI startup(): 5 678807 node02 {1,5,9,13,58,62} [0] MPI startup(): 6 678808 node02 {17,21,25,29,33,37} [0] MPI startup(): 7 678809 node02 {41,45,49,53,57,61} [0] MPI startup(): 8 678810 node02 {3,7,11,15,19,23} [0] MPI startup(): 9 678811 node02 {27,31,35,39,43,47} [0] MPI startup(): I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.4.0 [0] MPI startup(): I_MPI_MPIRUN=mpirun [0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc [0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm [0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default [0] MPI startup(): I_MPI_SHM_CMA=0 [0] MPI startup(): I_MPI_ADJUST_SCATTERV=0 [0] MPI startup(): I_MPI_DEBUG=30 starting wrf task 0 of 10 starting wrf task 1 of 10 starting wrf task 4 of 10 starting wrf task 6 of 10 starting wrf task 9 of 10 starting wrf task 2 of 10 starting wrf task 3 of 10 starting wrf task 5 of 10 starting wrf task 7 of 10 starting wrf task 8 of 10 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 1 PID 678803 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 2 PID 678804 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 3 PID 678805 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 4 PID 678806 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 5 PID 678807 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 6 PID 678808 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 7 PID 678809 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 8 PID 678810 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) =================================================================================== =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = RANK 9 PID 678811 RUNNING AT node02 = KILLED BY SIGNAL: 9 (Killed) ===================================================================================