I've previously identified this problem as I looked into poor network performance. In particular I'm using a private domain name for a specific private subnet. I've observed the MPI app using a much slower public subnet. The masks and access to the various hosts all look good to ping/traceroute. fi_info -n bubbles1.macaw.local returns eth1 for all but one host where it returns eth0.
I edited my mpiexec command line to specify this domain:
-host bubbles1 -env FI_SOCKETS_IFACE=eth0 -n 8 ${exec}
All the other hosts use
-genv FI_SOCKETS_IFACE=eth1
Performance looks a lot better when all the hosts are running on my private subnet (20gpbs, TCP). None of the usual tcp tools and configuration options revealed this problem.
Here's what my tcp/ip configuration looks like (the same on all hosts).
Ethernet adapter Team0:
Connection-specific DNS Suffix . : macaw.local
Description . . . . . . . . . . . : TEAM: Team0
Physical Address. . . . . . . . . : xx-xx-xx-xx-xx-xx
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.x.0.x(Preferred)
Subnet Mask . . . . . . . . . . . : 255.0.0.0
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled
So, do I have to re-compile the libfabric library to "fix" libfabric or is there another way to accomplish fixing its domain assignment?
This particular discovery was made using v2019.5. I had to back down from oneapi because of poor performance. I suspect that oneapi suffers from the same domain assignment problem. I'm going to try oneapi yet again.
TIA
Art
