Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2218 Discussions

MPI (Windows OS) internode synchronization issue?

NormanM
Beginner
‎12-02-2022 06:10 PM
1,584 Views

I'm having trouble getting an improvised Windows MPI cluster working.Intel MPI 2022.3
It appears to hang at places which require synchronization between the nodes. e.g. MPI_Barrier(), send/receive calls. My guess is it hangs waiting for a response which never comes? There might also be some element of race conditions in the mix, because for some scenarios messages seem to get through.

 

I have three computers (two desktops (<node0> intel CPU i9-7980xe, <node1> AMD Ryzen 7950X ) and 1 laptop (Intel CPU)). Mostly I've been testing the two desktops and the laptop was brought in for further debugging, but has the same issues.
<node0> is Microsoft Windows [Version 10.0.22621.819]
<node1> is Microsoft Windows [Version 10.0.22621.525]
The two desktops are fresh installs of Windows 11 (laptop is windows 8.1), firewall turned off, not much else on the machine except VIsual Studio 2019/2022 and CUDA 11.8.
These are consumer machines, so they only have ethernet. They have other adapters like bluetooth, and other NICs, but I've disabled all of those except the one connecting the two desktops. I've tried connecting via a switch, as well as back-to-back.

 

Things that work...
<hostname simple test>
Any computer can launch 'hostname' on any or all of the other nodes, including itself.

<IMB-MPI1 pingpong and test.c Intel example>
Multiple processes can be launched successfully on the same machine (either remotely, or on the local machine, as long as all the processes run on the same machine).
Using two different ethernet ports/IP address on the same machine, treating them as if they are separate hosts.
A bit surprisingly, launching a simple test that initialises MPI, and prints the processor name, world size and rank. However this doesn't require intercommunication between nodes.

Issues arise around scenarios involving MPI communication/syncronization between nodes.

Examples below with an example program, that simply sends a short 1024 byte buffer from one node to the next.
Scenario 1 : Local <node0> sends to remote <node1> (ok)
Hello world from processor <node1>, rank 1 out of 2 processors
<rank 1> Get array length...
<rank 1> Incoming buffer of length 1024
<rank 1> Copied buffer 1024
Hello world from processor <node0>, rank 0 out of 2 processors
<rank 0> sent buffer[1024]

Scenario 2 : Remote <node1> sends to local <node0> (hangs)
Hello world from processor <node1>, rank 1 out of 2 processors
<rank 1> sent buffer[1024]

Scenario 3 : Remote <node1> sends to local <node0> {same as above, but enumeration of ranks for the nodes is flipped} (hangs)
Hello world from processor <node1>, rank 0 out of 2 processors
<rank 0> sent buffer[1024]

Scenario 4 : Changing the MPI_Send call to an MPI_SSend, or adding an MPI_Barrier before the start of the send/receive causes a hard freeze which prints nothing.

Now if I flip the order of the nodes, so that I launch off the command prompt of <node1> instead of <node0>
Scenario 5 : Local <node1> sends to remote <node0> (ok)
Hello world from processor <node1>, rank 1 out of 2 processors
<rank 1> Get array length...
<rank 1> Incoming buffer of length 1024
<rank 1> Copied buffer 1024
Hello world from processor <node0>, rank 0 out of 2 processors
<rank 0> sent buffer[1024]

Scenario 6 : Remote <node0> sends to local <node1>
Hello world from processor <node1>, rank 0 out of 2 processors
<rank 0> sent buffer[1024]
Hello world from processor <node0>, rank 1 out of 2 processors
<rank 1> Get array length...
<1> Incoming buffer of length 1024
<1> Copied buffer 1024

Adding an MPI_Barrier() before the send/receive causes a crash. However, so does calling with I_MPI_DEBUG={big number}. As if there's some kind of race-type condition occurring?
DEBUG output visible below.

Also, if I try the fi_pingpong utility C:\Program Files (x86)\Intel\oneAPI\mpi\2021.7.0\libfabric\bin\utils\fi_pingpong.exe
This utility works, for either nodes being the server. No issues observed there.


<With MPI_Barrier call>

[mpiexec@node1] Launch arguments: C:\Program Files (x86)\Intel\oneAPI\mpi\latest\bin\\hydra_bstrap_proxy.exe --upstream-host 192.168.1.3 --upstream-port 49699 --pgid 0 --launcher service --launcher-number 0 --base-path C:\Program Files (x86)\Intel\oneAPI\mpi\latest\bin --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --service_port 0 --proxy-id 1 --node-id 1 --subtree-size 1 --upstream-fd 684 C:\Program Files (x86)\Intel\oneAPI\mpi\latest\bin\\hydra_pmi_proxy.exe --usize -1 --auto-cleanup 1 --abort-signal 9
[proxy:0:1@node1] pmi cmd from fd 536: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@node1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@node1] pmi cmd from fd 536: cmd=get_maxes
[proxy:0:1@node1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@node1] pmi cmd from fd 536: cmd=get_appnum
[proxy:0:1@node1] PMI response: cmd=appnum appnum=0
[proxy:0:1@node1] pmi cmd from fd 536: cmd=get_my_kvsname
[proxy:0:1@node1] PMI response: cmd=my_kvsname kvsname=kvs_14804_0
[proxy:0:1@node1] pmi cmd from fd 536: cmd=get kvsname=kvs_14804_0 key=PMI_process_mapping
[proxy:0:1@node1] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1))
[proxy:0:1@node1] pmi cmd from fd 536: cmd=barrier_in
[proxy:0:0@node0] pmi cmd from fd 524: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@node0] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@node0] pmi cmd from fd 524: cmd=get_maxes
[proxy:0:0@node0] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@node0] pmi cmd from fd 524: cmd=get_appnum
[proxy:0:0@node0] PMI response: cmd=appnum appnum=0
[proxy:0:0@node0] pmi cmd from fd 524: cmd=get_my_kvsname
[proxy:0:0@node0] PMI response: cmd=my_kvsname kvsname=kvs_14804_0
[0] MPI startup(): Run 'pmi_process_mapping' nodemap algorithm
[proxy:0:0@node0] pmi cmd from fd 524: cmd=get kvsname=kvs_14804_0 key=PMI_process_mapping
[proxy:0:0@node0] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1))
[0] MPI startup(): Intel(R) MPI Library, Version 2021.7 Build 20220909
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[proxy:0:0@node0] pmi cmd from fd 524: cmd=barrier_in
[proxy:0:1@node1] PMI response: cmd=barrier_out
[proxy:0:0@node0] PMI response: cmd=barrier_out
libfabric:14696:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:14696:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:14696:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ZE not supported
libfabric:14696:core:mr:ofi_default_cache_size():78<info> default cache size=2097152
libfabric:14696:core:core:ofi_register_provider():474<info> registering provider: netdir (113.20)
libfabric:14696:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
libfabric:14696:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:14696:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:14696:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_perf (113.20)
libfabric:14696:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_noop (113.20)
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:fi_getinfo():1123<warn> Can't find provider with the highest priority
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:14696:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:14696:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:14696:core:core:fi_getinfo():1201<info> Start regular provider search because provider with the highest priority tcp can not be initialized
libfabric:14696:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:14696:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:14696:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:14696:netdir:core:ofi_nd_startup():601<info> ofi_nd_startup: starting initialization
libfabric:14696:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:14696:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_DGRAM
libfabric:14696:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:14696:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:14696:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:14696:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:14696:sockets:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:sockets:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:sockets:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:sockets:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:14696:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:14696:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:14696:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:14696:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:14696:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:14696:tcp:core:ofi_check_rx_attr():786<info> Tx only caps ignored in Rx caps
libfabric:14696:tcp:core:ofi_check_tx_attr():884<info> Rx only caps ignored in Tx caps
libfabric:14696:ofi_rxm:av:util_verify_av_attr():508<warn> Shared AV is unsupported
libfabric:1[proxy:0:1@node1] pmi cmd from fd 536: cmd=put kvsname=kvs_14804_0 key=bc-1 value=mpi#0200C22DC0A801030000000000000000$
[proxy:0:1@node1] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@node1] pmi cmd from fd 536: cmd=barrier_in
[0] MPI startup(): libfabric version: 1.13.2-impi
libfabric:15072:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:15072:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:15072:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ZE not supported
libfabric:15072:core:mr:ofi_default_cache_size():78<info> default cache size=1864135
libfabric:15072:core:core:ofi_register_provider():474<info> registering provider: netdir (113.20)
libfabric:15072:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
libfabric:15072:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:15072:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:15072:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_perf (113.20)
libfabric:15072:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_noop (113.20)
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:ofi_layering_ok():1001[0] MPI startup(): max_ch4_vnis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 0
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
<info> Need core provider, skipping ofi_rxm
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:fi_getinfo():1123<warn> Can't find provider with the highest priority
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6:[0] MPI startup(): libfabric provider: tcp;ofi_rxm
[0] MPI startup(): detected tcp;ofi_rxm provider, set device name to "tcp-ofi-rxm"
//[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:15072:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:15072:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:15072:core:core:fi_getinfo():1201<info> Start regular provider search because provider with the highest priority tcp can not be initialized
libfabric:15072:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:15072:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:15072:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:15072:netdir:core:ofi_nd_startup():601<info> ofi_nd_startup: starting initialization
libfabric:15072:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:15072:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_DGRAM
libfabric:15072:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:15072:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:15072:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:15072:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:15072:sockets:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:sockets:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:sockets:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:sockets:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:15072:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:15072:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:15072:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:15072:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:15072:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:15072:tcp:core:ofi_check_rx_attr():786<info> Tx only caps ignored in Rx caps
libfabric:15072:tcp:core:ofi_check_tx_attr():884<info> Rx only caps ignored in Tx caps
libfabric:15072:ofi_rxm:av:util_verify_av_attr():508<warn> Shared AV is un[0] MPI startup(): addrnamelen: zu
[proxy:0:0@node0] pmi cmd from fd 524: cmd=put kvsname=kvs_14804_0 key=bc-0 value=mpi#0200C24FC0A801020000000000000000$
[proxy:0:0@node0] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@node0] pmi cmd from fd 524: cmd=barrier_in
[proxy:0:1@node1] PMI response: cmd=barrier_out
[proxy:0:1@node1] pmi cmd from fd 536: cmd=get kvsname=kvs_14804_0 key=bc-0
[proxy:0:0@node0] PMI response: cmd=barrier_out
[proxy:0:1@node1] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C24FC0A801020000000000000000$
[proxy:0:0@node0] pmi cmd from fd 524: cmd=get kvsname=kvs_14804_0 key=bc-0
[proxy:0:1@node1] pmi cmd from fd 536: cmd=get kvsname=kvs_14804_0 key=bc-1
[proxy:0:0@node0] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C24FC0A801020000000000000000$
[proxy:0:1@node1] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C22DC0A801030000000000000000$
[proxy:0:0@node0] pmi cmd from fd 524: cmd=get kvsname=kvs_14804_0 key=bc-1
[1] MPI startup(): selected platform: unknown
[proxy:0:0@node0] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C22DC0A801030000000000000000$
[1] MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): selected platform: unknown
[0] MPI startup(): File "/tuning_skx_shm-ofi_tcp-ofi-rxm.dat" not found
[0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): Looking for tuning file: "/tuning_generic_shm-ofi_tcp-ofi-rxm.dat"
[0] MPI startup(): Looking for tuning file: "/tuning_generic_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for ch4 level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for net level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for shm level
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): threading: num_pools: 1
[0] MPI startup(): threading: enable_sep: 0
[0] MPI startup(): threading: direct_recv: 1
[0] MPI startup(): threading: zero_op_flags: 0
[0] MPI startup(): threading: num_am_buffers: 8
[0] MPI startup(): tag bits available: 19 (TAG_UB value: 524287)
[0] MPI startup(): source bits available: 20 (Maximal number of rank: 1048575)
[0] MPI startup(): Imported environment partly inaccesible. Map=0 Info=0


<Without MPI_Barrier call>

[mpiexec@node1] Launch arguments: C:\Program Files (x86)\Intel\oneAPI\mpi\latest\bin\\hydra_bstrap_proxy.exe --upstream-host 192.168.1.3 --upstream-port 49752 --pgid 0 --launcher service --launcher-number 0 --base-path C:\Program Files (x86)\Intel\oneAPI\mpi\latest\bin --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --service_port 0 --proxy-id 1 --node-id 1 --subtree-size 1 --upstream-fd 684 C:\Program Files (x86)\Intel\oneAPI\mpi\latest\bin\\hydra_pmi_proxy.exe --usize -1 --auto-cleanup 1 --abort-signal 9
[proxy:0:1@node1] pmi cmd from fd 524: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@node1] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@node1] pmi cmd from fd 524: cmd=get_maxes
[proxy:0:1@node1] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@node1] pmi cmd from fd 524: cmd=get_appnum
[proxy:0:1@node1] PMI response: cmd=appnum appnum=0
[proxy:0:1@node1] pmi cmd from fd 524: cmd=get_my_kvsname
[proxy:0:1@node1] PMI response: cmd=my_kvsname kvsname=kvs_8268_0
[proxy:0:1@node1] pmi cmd from fd 524: cmd=get kvsname=kvs_8268_0 key=PMI_process_mapping
[proxy:0:1@node1] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1))
[proxy:0:1@node1] pmi cmd from fd 524: cmd=barrier_in
[proxy:0:0@node0] pmi cmd from fd 520: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@node0] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@node0] pmi cmd from fd 520: cmd=get_maxes
[proxy:0:0@node0] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@node0] pmi cmd from fd 520: cmd=get_appnum
[proxy:0:0@node0] PMI response: cmd=appnum appnum=0
[proxy:0:0@node0] pmi cmd from fd 520: cmd=get_my_kvsname
[proxy:0:0@node0] PMI response: cmd=my_kvsname kvsname=kvs_8268_0
[0] MPI startup(): Run 'pmi_process_mapping' nodemap algorithm
[proxy:0:0@node0] pmi cmd from fd 520: cmd=get kvsname=kvs_8268_0 key=PMI_process_mapping
[proxy:0:0@node0] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1))
[0] MPI startup(): Intel(R) MPI Library, Version 2021.7 Build 20220909
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
[proxy:0:0@node0] pmi cmd from fd 520: cmd=barrier_in
[proxy:0:1@node1] PMI response: cmd=barrier_out
[proxy:0:0@node0] PMI response: cmd=barrier_out
libfabric:11512:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:11512:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:11512:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ZE not supported
libfabric:11512:core:mr:ofi_default_cache_size():78<info> default cache size=2097152
libfabric:11512:core:core:ofi_register_provider():474<info> registering provider: netdir (113.20)
libfabric:11512:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
libfabric:11512:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:11512:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:11512:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_perf (113.20)
libfabric:11512:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_noop (113.20)
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:fi_getinfo():1123<warn> Can't find provider with the highest priority
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:11512:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:11512:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:11512:core:core:fi_getinfo():1201<info> Start regular provider search because provider with the highest priority tcp can not be initialized
libfabric:11512:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:11512:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:11512:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:11512:netdir:core:ofi_nd_startup():601<info> ofi_nd_startup: starting initialization
libfabric:11512:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:11512:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_DGRAM
libfabric:11512:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:11512:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:11512:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:11512:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:11512:sockets:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:sockets:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:sockets:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:sockets:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.3, speed 1000000000
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:11512:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:11512:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:11512:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.3, iface name: eth0, speed: 1000000000
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:11512:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:11512:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:11512:tcp:core:ofi_check_rx_attr():786<info> Tx only caps ignored in Rx caps
libfabric:11512:tcp:core:ofi_check_tx_attr():884<info> Rx only caps ignored in Tx caps
libfabric:11512:ofi_rxm:av:util_verify_av_attr():508<warn> Shared AV is unsupported
libfabric:1[proxy:0:1@node1] pmi cmd from fd 524: cmd=put kvsname=kvs_8268_0 key=bc-1 value=mpi#0200C262C0A801030000000000000000$
[proxy:0:1@node1] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@node1] pmi cmd from fd 524: cmd=barrier_in
[0] MPI startup(): libfabric version: 1.13.2-impi
libfabric:16136:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_CUDA not supported
libfabric:16136:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ROCR not supported
libfabric:16136:core:core:ofi_hmem_init():209<info> Hmem iface FI_HMEM_ZE not supported
libfabric:16136:core:mr:ofi_default_cache_size():78<info> default cache size=1864135
libfabric:16136:core:core:ofi_register_provider():474<info> registering provider: netdir (113.20)
libfabric:16136:core:core:ofi_register_provider():474<info> registering provider: ofi_rxm (113.20)
libfabric:16136:core:core:ofi_register_provider():474<info> registering provider: sockets (113.20)
libfabric:16136:core:core:ofi_register_provider():474<info> registering provider: tcp (113.20)
libfabric:16136:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_perf (113.20)
libfabric:16136:core:core:ofi_register_provider():474<info> registering provider: ofi_hook_noop (113.20)
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:ofi_layering_ok():1001[0] MPI startup(): max_ch4_vnis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 0
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
<info> Need core provider, skipping ofi_rxm
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:fi_getinfo():1123<warn> Can't find provider with the highest priority
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:core:core:ofi_layering_ok():1001<info> Need core provider, skipping ofi_rxm
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6:[0] MPI startup(): libfabric provider: tcp;ofi_rxm
[0] MPI startup(): detected tcp;ofi_rxm provider, set device name to "tcp-ofi-rxm"
//[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:16136:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:16136:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:16136:core:core:fi_getinfo():1201<info> Start regular provider search because provider with the highest priority tcp can not be initialized
libfabric:16136:tcp:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:16136:tcp:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:16136:tcp:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:16136:netdir:core:ofi_nd_startup():601<info> ofi_nd_startup: starting initialization
libfabric:16136:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:16136:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_DGRAM
libfabric:16136:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:16136:sockets:core:ofi_check_ep_type():658<info> unsupported endpoint type
libfabric:16136:sockets:core:ofi_check_ep_type():659<info> Supported: FI_EP_MSG
libfabric:16136:sockets:core:ofi_check_ep_type():659<info> Requested: FI_EP_RDM
libfabric:16136:sockets:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:sockets:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:sockets:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:sockets:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:tcp:core:util_getinfo_ifs():318<info> Chosen addr for using: 192.168.1.2, speed 10000000000
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:16136:core:core:fi_fabric():1423<info> Opened fabric: 192.168.1.0/24
libfabric:16136:core:core:fi_getinfo():1138<info> Found provider with the highest priority tcp, must_use_util_prov = 1
libfabric:16136:tcp:core:ofi_get_list_of_addr():1776<info> Available addr: 192.168.1.2, iface name: eth0, speed: 10000000000
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1619<info> available addr: : fi_sockaddr_in://127.0.0.1:0
libfabric:16136:tcp:core:ofi_insert_loopback_addr():1634<info> available addr: : fi_sockaddr_in6://[::1]:0
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, netdir has been skipped. To use netdir, please, set FI_PROVIDER=netdir
libfabric:16136:core:core:fi_getinfo():1161<info> Since tcp can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets
libfabric:16136:tcp:core:ofi_check_rx_attr():786<info> Tx only caps ignored in Rx caps
libfabric:16136:tcp:core:ofi_check_tx_attr():884<info> Rx only caps ignored in Tx caps
libfabric:16136:ofi_rxm:av:util_verify_av_attr():508<warn> Shared AV is un[0] MPI startup(): addrnamelen: zu
[proxy:0:0@node0] pmi cmd from fd 520: cmd=put kvsname=kvs_8268_0 key=bc-0 value=mpi#0200C27DC0A801020000000000000000$
[proxy:0:0@node0] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@node0] pmi cmd from fd 520: cmd=barrier_in
[proxy:0:1@node1] PMI response: cmd=barrier_out
[proxy:0:1@node1] pmi cmd from fd 524: cmd=get kvsname=kvs_8268_0 key=bc-0
[proxy:0:0@node0] PMI response: cmd=barrier_out
[proxy:0:1@node1] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C27DC0A801020000000000000000$
[proxy:0:0@node0] pmi cmd from fd 520: cmd=get kvsname=kvs_8268_0 key=bc-0
[proxy:0:1@node1] pmi cmd from fd 524: cmd=get kvsname=kvs_8268_0 key=bc-1
[proxy:0:0@node0] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C27DC0A801020000000000000000$
[proxy:0:1@node1] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C262C0A801030000000000000000$
[proxy:0:0@node0] pmi cmd from fd 520: cmd=get kvsname=kvs_8268_0 key=bc-1
[1] MPI startup(): selected platform: unknown
[proxy:0:0@node0] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200C262C0A801030000000000000000$
[1] MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
[0] MPI startup(): selected platform: unknown
[0] MPI startup(): File "/tuning_skx_shm-ofi_tcp-ofi-rxm.dat" not found
[0] MPI startup(): Load tuning file: "/tuning_skx_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): Looking for tuning file: "/tuning_generic_shm-ofi_tcp-ofi-rxm.dat"
[0] MPI startup(): Looking for tuning file: "/tuning_generic_shm-ofi.dat"
[0] MPI startup(): File "/tuning_skx_shm-ofi.dat" not found
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for ch4 level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for net level
[0] MPI startup(): File "" not found
[0] MPI startup(): Unable to read tuning file for shm level
[0] MPI startup(): threading: mode: direct
[0] MPI startup(): threading: vcis: 1
[0] MPI startup(): threading: app_threads: -1
[0] MPI startup(): threading: runtime: generic
[0] MPI startup(): threading: progress_threads: 0
[0] MPI startup(): threading: async_progress: 0
[0] MPI startup(): threading: lock_level: global
[0] MPI startup(): threading: num_pools: 1
[0] MPI startup(): threading: enable_sep: 0
[0] MPI startup(): threading: direct_recv: 1
[0] MPI startup(): threading: zero_op_flags: 0
[0] MPI startup(): threading: num_am_buffers: 8
[0] MPI startup(): tag bits available: 19 (TAG_UB value: 524287)
[0] MPI startup(): source bits available: 20 (Maximal number of rank: 1048575)
[0] MPI startup(): Imported environment partly inaccesible. Map=0 Info=0

Labels (1)
Labels:
0 Kudos

Link Copied

0 Replies
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.