Intel MPI mlx provider issue

[Antonio_D]

I have a program that I have compiled with oneAPI 2025.1 that will run just fine using I_MPI_OFI_PROVIDER=verbs (or any other provider really), but will not run with I_MPI_OFI_PROVIDER=mlx.

I_MPI_DEBUG=30 output:

[0] MPI startup(): PMI API: pmix
[0] MPI startup(): PMIx version: OpenPMIx 5.0.7 (PMIx Standard: 5.1, Stable ABI: 5.0, Provisional ABI: 5.0)
[0] MPI startup(): Intel(R) MPI Library, Version 2021.15  Build 20250213 (id: d233448)
[0] MPI startup(): Copyright (C) 2003-2025 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
[0] MPI startup(): libfabric loaded: libfabric.so.1 
[0] MPI startup(): libfabric version: 1.21.0-impi
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: verbs (121.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "verbs" filtered by provider include/exclude list, skipping
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: verbs (121.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "verbs" filtered by provider include/exclude list, skipping
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: tcp (121.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "tcp" filtered by provider include/exclude list, skipping
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: shm (200.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "shm" filtered by provider include/exclude list, skipping
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: ofi_rxm (121.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: psm2 (121.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "psm2" filtered by provider include/exclude list, skipping
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: psm3 (707.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "psm3" filtered by provider include/exclude list, skipping
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: mlx (1.4)
libfabric:1780409:1744907435::core:core:ofi_reg_dl_prov():675<warn> dlopen(/projects/site/gred/smpg/software/oneAPI/2025.1/mpi/2021.15/opt/mpi/libfabric/lib/prov/libefa-fi.so): libefa.so.1: cannot open shared object file: No such file or directory
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: ofi_hook_noop (121.0)
libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: off_coll (121.0)
libfabric:1780409:1744907435::core:core:fi_getinfo_():1449<info> Found provider with the highest priority mlx, must_use_util_prov = 0
[0] MPI startup(): max_ch4_vnis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 0
[0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1)
libfabric:1780409:1744907435::core:core:fi_getinfo_():1449<info> Found provider with the highest priority mlx, must_use_util_prov = 0
[0] MPI startup(): libfabric provider: mlx
libfabric:1780409:1744907435::core:core:fi_fabric_():1745<info> Opened fabric: mlx
libfabric:1780409:1744907435::core:core:fi_fabric_():1756<info> Using mlx provider 1.21, path:/projects/site/gred/smpg/software/oneAPI/2025.1/mpi/2021.15/opt/mpi/libfabric/lib/prov/libmlx-fi.so
[0] MPI startup(): addrnamelen: 1024
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(196)........: 
MPID_Init(1719)..............: 
MPIDI_OFI_mpi_init_hook(1741): 
MPIDU_bc_table_create(340)...: Missing hostname or invalid host/port description in business card
Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(196)........: 
MPID_Init(1719)..............: 
MPIDI_OFI_mpi_init_hook(1741): 
MPIDU_bc_table_create(340)...: Missing hostname or invalid host/port description in business card
slurmstepd: error: *** STEP 16814.0 ON sc1nc124 CANCELLED AT 2025-04-17T09:30:36 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: sc1nc124: tasks 0-11: Killed

ucx_info -v:

# Library version: 1.16.0
# Library path: /usr/lib64/libucs.so.0
# API headers version: 1.16.0
# Git branch '', revision 02432d3
# Configured with: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --enable-mt --disable-params-check --without-go --without-java --enable-cma --with-cuda --with-gdrcopy --with-verbs --with-knem --with-rdmacm --without-rocm --with-xpmem --without-fuse3 --without-ugni --with-cuda=/usr/local/cuda-12.2

ucx_info -d | grep Transport:

#      Transport: self
#      Transport: tcp
#      Transport: tcp
#      Transport: tcp
#      Transport: sysv
#      Transport: posix
#      Transport: dc_mlx5
#      Transport: rc_verbs
#      Transport: rc_mlx5
#      Transport: ud_verbs
#      Transport: ud_mlx5
#      Transport: cma
#      Transport: xpmem

mlx looks like it should be available from all of the outputs I see.  This looks like a cluster configuration issue, but I don't know where to start troubleshooting.  SLURM job scheduler is in use.

[TobiasK]

@Antonio_D 

Can you please provide the output of

fi_info

and also (without setting FI_PROVIDER)

I_MPI_DEBUG=30 I_MPI_HYDRA_DEBUG=1 mpirun IMB-MPI1 allreduce

[Antonio_D]

See the attached files.  We have recently found that the mlx provider works properly with mpirun, but not srun (when submitting to SLURM using an sbatch script).

[TobiasK]

@Antonio_D 

thanks can you please also attach the output of the failing srun command, e.g.:

I_MPI_DEBUG=30 I_MPI_HYDRA_DEBUG=1 srun IMB-MPI1 allreduce

[Antonio_D]

See attached.  When I run it with pmix using the following:

I_MPI_DEBUG=30 I_MPI_HYDRA_DEBUG=1 I_MPI_PMI_LIBRARY=/.../software/pmix/lib/libpmix.so srun --mpi=pmix IMB-MPI1 allreduce

I get the error, also attached.

[TobiasK]

Can you please also provide the log with I_MPI_DEBUG=1000 ?

[Antonio_D]

See attached for files with I_MPI_DEBUG=1000

[TobiasK]

@Antonio_D 
can you please try to set

I_MPI_HYDRA_IFACE=ib0

Also since it is working with mpirun, is slurm correctly setup, e.g. are you setting:

PropagateResourceLimitsExcept=MEMLOCK 

https://slurm.schedmd.com/faq.html#memlock

[Antonio_D]

Same result with I_MPI_HYDRA_IFACE=ib0