UCX errors when running MPI

[kwjung]

I was getting a UCX error when trying to run a program on MPI.

To get more information, I ran it with I_MPI_DEBUG set to 10 and got the following message

[0] MPI startup(): Intel(R) MPI Library, Version 2021.8  Build 20221129 (id: 339ec755a1)
[0] MPI startup(): Copyright (C) 2003-2022 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): shm segment size (811 MB per rank) * (4 local ranks) = 3245 MB total
[0] MPI startup(): libfabric version: 1.13.2rc1-impi
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
[0] MPI startup(): libfabric provider: mlx
[1719476438.048014] [duecn27:740870:0]         select.c:450  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable
[1719476438.048011] [duecn27:740871:0]         select.c:450  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable
[1719476438.048004] [duecn27:740872:0]         select.c:450  UCX  ERROR no active messages transport to <no debug data>: self/memory - Destination is unreachable

It appears that an error occurred in MPI before entering the program.

Please let me know how can I resolve this problem

[TobiasK]

@kwjung 

On the forum, I can only help you with issues you encounter with the latest public release, that is 2021.13.
Anyway, the UCX messages indicates that your UCX setup is incomplete.
Please make sure that you installed the latest UCX stack. If it still does not work, please have a look at the output of ucx_info -d and make sure the Infiniband NIC is listed.

[kwjung]

Hello TobiasK,

Thank you for reply and it is difficult to change to the latest version of MPI now. 

Is there a guide or troubleshooting documentation that I can refer to regarding UCX setup?