Intel® MPI Library
Get help with building, analyzing, optimizing, and scaling high-performance computing (HPC) applications.
2154 Discussions

FI_LOG_LEVEL=debug shows wrong IB interface is used and mpirun fails across multiple nodes

RD1
Novice
639 Views

Hello,

We have installed some new nodes with multiple IB interfaces (mlx5_0 is used for ethernet, mlx5_1 is not used and mlx5_2 is connected to the IB fabric), however when we run the intel benchmarks with FI_LOG_LEVEL=debug set we see that mlx5_0 is chosen (instead of mlx5_2) as you can see from:

 

$ echo $LD_LIBRARY_PATH
/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/libfabric/lib:/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/lib/release:/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/lib:/u/home/systems/dauria/torch/install/lib:/u/home/systems/dauria/torch/install/lib
$ /u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/bin/mpirun -genv LD_LIBRARY_PATH=$LD_LIBRARY_PATH -genv FI_LOG_LEVEL=debug -n 4 -ppn 2 -hosts n1018,n1019 IMB-MPI1 allreduce -iter 1000,800 -time 4800 2>&1 | grep mlx5_
libfabric:47326:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:47325:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:47326:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:47325:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31436:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31435:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31436:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0
libfabric:31435:mlx:core:mlx_getinfo():211<info> primary detected device: mlx5_0


...

 

 This is an issue when running across nodes that have a different arrangement of IB cards (mlx5_0 used for IB).

On the node with the three IB interfaces, we have:

 

$ ibstat
CA 'mlx5_0'
	CA type: MT4117
	Number of ports: 1
	Firmware version: 14.28.1300
	Hardware version: 0
	Node GUID: 0x7cc25515d93fcdb6
	System image GUID: 0x7cc25515d93fcdb6
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 25
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x00010000
		Port GUID: 0x7ec255fffe3fcdb6
		Link layer: Ethernet
CA 'mlx5_1'
	CA type: MT4117
	Number of ports: 1
	Firmware version: 14.28.1300
	Hardware version: 0
	Node GUID: 0x7cc25515d93fcdb7
	System image GUID: 0x7cc25515d93fcdb6
	Port 1:
		State: Down
		Physical state: Disabled
		Rate: 40
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x00010000
		Port GUID: 0x7ec255fffe3fcdb7
		Link layer: Ethernet
CA 'mlx5_2'
	CA type: MT4115
	Number of ports: 1
	Firmware version: 12.28.2006
	Hardware version: 0
	Node GUID: 0x0cc47affff602ecc
	System image GUID: 0x0cc47affff602ecc
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 100
		Base lid: 443
		LMC: 0
		SM lid: 4
		Capability mask: 0x2651e848
		Port GUID: 0x0cc47affff602ecc
		Link layer: InfiniBand

 

and:

 

$ fi_info -l
mlx:
    version: 1.4
psm3:
    version: 1104.0
ofi_rxm:
    version: 113.20
verbs:
    version: 113.20
tcp:
    version: 113.20
sockets:
    version: 113.20
shm:
    version: 116.10
ofi_hook_noop:
    version: 113.20

 

and:

 

$ /u/local/apps/ucx/1.12/bin/ucx_info -d | grep -i tran
#      Transport: posix
#      Transport: sysv
#      Transport: self
#      Transport: tcp
#      Transport: tcp
#      Transport: rc_verbs
#      Transport: rc_mlx5
#      Transport: ud_verbs
#      Transport: ud_mlx5
#      Transport: rc_verbs
#      Transport: rc_mlx5
#      Transport: dc_mlx5
#      Transport: ud_verbs
#      Transport: ud_mlx5
#      Transport: cma

 

 The node with two interfaces that has mlx5_0 connected to IB:

 

$ ibstat
CA 'mlx5_0'
	CA type: MT4119
	Number of ports: 1
	Firmware version: 16.34.1002
	Hardware version: 0
	Node GUID: 0x88e9a4ffff605532
	System image GUID: 0x88e9a4ffff605532
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 100
		Base lid: 626
		LMC: 0
		SM lid: 4
		Capability mask: 0xa651e848
		Port GUID: 0x88e9a4ffff605532
		Link layer: InfiniBand
CA 'mlx5_1'
	CA type: MT4119
	Number of ports: 1
	Firmware version: 16.34.1002
	Hardware version: 0
	Node GUID: 0x88e9a4ffff605533
	System image GUID: 0x88e9a4ffff605532
	Port 1:
		State: Down
		Physical state: Polling
		Rate: 10
		Base lid: 65535
		LMC: 0
		SM lid: 0
		Capability mask: 0xa651e848
		Port GUID: 0x88e9a4ffff605533
		Link layer: InfiniBand

 

Is it possible to specify I_MPI_OFA_ADAPTER_NAME to force intelmpi to pick the correct IB interface? This env var does no longer seem to be documented/available.

Note mpich and openmpi seem able to pick the correct interface the problem appears to be related to intel mpi.

We are running CentOS 7.9.2009

Please let me know if there is any further detail I can provide.

 

Any help would be appreciated, thanks in advance!

0 Kudos
3 Replies
VarshaS_Intel
Moderator
598 Views

Hi,

 

Thanks for posting in Intel Communities.

 

You can try adding the environment variable "UCX_NET_DEVICES" to select from the available adapters as shown in the below command:

 

/u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/bin/mpirun -genv LD_LIBRARY_PATH=$LD_LIBRARY_PATH -genv FI_LOG_LEVEL=debug -n 4 -ppn 2 -env UCX_NET_DEVICES=mlx5_2:1 -hosts n1018,n1019 IMB-MPI1 allreduce -iter 1000,800 -time 4800 2>&1 | grep mlx5_

 

You can also check UCX_LOG_LEVEL=info, to check which adapter is being used.

UCX_LOG_LEVEL=info /u/local/compilers/intel/oneapi/2023/mpi/2021.8.0/bin/mpirun -genv LD_LIBRARY_PATH=$LD_LIBRARY_PATH -genv FI_LOG_LEVEL=debug -n 4 -ppn 2 -env UCX_NET_DEVICES=<required_interface> -hosts n1018,n1019 IMB-MPI1 allreduce -iter 1000,800 -time 4800 2>&1 | grep mlx5_

Could you please try and let us know if you are having any other issues?

 

Thanks & Regards,

Varsha

 

0 Kudos
VarshaS_Intel
Moderator
573 Views

Hi,


We have not heard back from you. Could you please provide us with an update on your issue?


Thanks & Regards,

Varsha


0 Kudos
VarshaS_Intel
Moderator
560 Views

Hi,

 

We have not heard back from you. We assume that your issue is resolved. This thread will no longer be monitored by Intel. If you need additional information, please start a new question.

 

Have a good day!

 

Thanks & Regards,

Varsha

 

0 Kudos
Reply