- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
We have installed some new nodes with multiple IB interfaces (mlx5_0 is used for ethernet, mlx5_1 is not used and mlx5_2 is connected to the IB fabric), however when we run the intel benchmarks with FI_LOG_LEVEL=debug set we see that mlx5_0 is chosen (instead of mlx5_2) as you can see from:
$ echo $LD_LIBRARY_PATH
/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/libfabric/lib:/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/lib/release:/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/lib:/u/home/systems/dauria/torch/install/lib:/u/home/systems/dauria/torch/install/lib
$ /u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/bin/mpirun -genv LD_LIBRARY_PATH=$LD_LIBRARY_PATH -genv FI_LOG_LEVEL=debug -n 4 -ppn 2 -hosts n1018,n1019 IMB-MPI1 allreduce -iter 1000,800 -time 4800 2>&1 | grep mlx5_
libfabric:47326:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:47325:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:47326:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:47325:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31436:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31435:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31436:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31435:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
...
This is an issue when running across nodes that have a different arrangement of IB cards (mlx5_0 used for IB).
On the node with the three IB interfaces, we have:
$ ibstat
CA 'mlx5_0'
CA type: MT4117
Number of ports: 1
Firmware version: 14.28.1300
Hardware version: 0
Node GUID: 0x7cc25515d93fcdb6
System image GUID: 0x7cc25515d93fcdb6
Port 1:
State: Active
Physical state: LinkUp
Rate: 25
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x7ec255fffe3fcdb6
Link layer: Ethernet
CA 'mlx5_1'
CA type: MT4117
Number of ports: 1
Firmware version: 14.28.1300
Hardware version: 0
Node GUID: 0x7cc25515d93fcdb7
System image GUID: 0x7cc25515d93fcdb6
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x7ec255fffe3fcdb7
Link layer: Ethernet
CA 'mlx5_2'
CA type: MT4115
Number of ports: 1
Firmware version: 12.28.2006
Hardware version: 0
Node GUID: 0x0cc47affff602ecc
System image GUID: 0x0cc47affff602ecc
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 443
LMC: 0
SM lid: 4
Capability mask: 0x2651e848
Port GUID: 0x0cc47affff602ecc
Link layer: InfiniBand
and:
$ fi_info -l
mlx:
version: 1.4
psm3:
version: 1104.0
ofi_rxm:
version: 113.20
verbs:
version: 113.20
tcp:
version: 113.20
sockets:
version: 113.20
shm:
version: 116.10
ofi_hook_noop:
version: 113.20
and:
$ /u/local/apps/ucx/1.12/bin/ucx_info -d | grep -i tran
# Transport: posix
# Transport: sysv
# Transport: self
# Transport: tcp
# Transport: tcp
# Transport: rc_verbs
# Transport: rc_mlx5
# Transport: ud_verbs
# Transport: ud_mlx5
# Transport: rc_verbs
# Transport: rc_mlx5
# Transport: dc_mlx5
# Transport: ud_verbs
# Transport: ud_mlx5
# Transport: cma
The node with two interfaces that has mlx5_0 connected to IB:
$ ibstat
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.34.1002
Hardware version: 0
Node GUID: 0x88e9a4ffff605532
System image GUID: 0x88e9a4ffff605532
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 626
LMC: 0
SM lid: 4
Capability mask: 0xa651e848
Port GUID: 0x88e9a4ffff605532
Link layer: InfiniBand
CA 'mlx5_1'
CA type: MT4119
Number of ports: 1
Firmware version: 16.34.1002
Hardware version: 0
Node GUID: 0x88e9a4ffff605533
System image GUID: 0x88e9a4ffff605532
Port 1:
State: Down
Physical state: Polling
Rate: 10
Base lid: 65535
LMC: 0
SM lid: 0
Capability mask: 0xa651e848
Port GUID: 0x88e9a4ffff605533
Link layer: InfiniBand
Is it possible to specify I_MPI_OFA_ADAPTER_NAME to force intelmpi to pick the correct IB interface? This env var does no longer seem to be documented/available.
Note mpich and openmpi seem able to pick the correct interface the problem appears to be related to intel mpi.
We are running CentOS 7.9.2009
Please let me know if there is any further detail I can provide.
Any help would be appreciated, thanks in advance!
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Thanks for posting in Intel Communities.
You can try adding the environment variable "UCX_NET_DEVICES" to select from the available adapters as shown in the below command:
/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/bin/mpirun -genv LD_LIBRARY_PATH=$LD_LIBRARY_PATH -genv FI_LOG_LEVEL=debug -n 4 -ppn 2 -env UCX_NET_DEVICES=mlx5_2:1 -hosts n1018,n1019 IMB-MPI1 allreduce -iter 1000,800 -time 4800 2>&1 | grep mlx5_
You can also check UCX_LOG_LEVEL=info, to check which adapter is being used.
UCX_LOG_LEVEL=info /u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/bin/mpirun -genv LD_LIBRARY_PATH=$LD_LIBRARY_PATH -genv FI_LOG_LEVEL=debug -n 4 -ppn 2 -env UCX_NET_DEVICES=<required_interface> -hosts n1018,n1019 IMB-MPI1 allreduce -iter 1000,800 -time 4800 2>&1 | grep mlx5_
Could you please try and let us know if you are having any other issues?
Thanks & Regards,
Varsha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
We have not heard back from you. Could you please provide us with an update on your issue?
Thanks & Regards,
Varsha
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
We have not heard back from you. We assume that your issue is resolved. This thread will no longer be monitored by Intel. If you need additional information, please start a new question.
Have a good day!
Thanks & Regards,
Varsha
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page