Intel MPI mlx provider issue

Antonio_D — Thu, 17 Apr 2025 18:27:39 GMT

I have a program that I have compiled with oneAPI 2025.1 that will run just fine using I_MPI_OFI_PROVIDER=verbs (or any other provider really), but will not run with I_MPI_OFI_PROVIDER=mlx.

I_MPI_DEBUG=30 output:

[0] MPI startup(): PMI API: pmix [0] MPI startup(): PMIx version: OpenPMIx 5.0.7 (PMIx Standard: 5.1, Stable ABI: 5.0, Provisional ABI: 5.0) [0] MPI startup(): Intel(R) MPI Library, Version 2021.15 Build 20250213 (id: d233448) [0] MPI startup(): Copyright (C) 2003-2025 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric loaded: libfabric.so.1 [0] MPI startup(): libfabric version: 1.21.0-impi libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: verbs (121.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "verbs" filtered by provider include/exclude list, skipping libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: verbs (121.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "verbs" filtered by provider include/exclude list, skipping libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: tcp (121.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "tcp" filtered by provider include/exclude list, skipping libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: shm (200.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "shm" filtered by provider include/exclude list, skipping libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: ofi_rxm (121.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: psm2 (121.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "psm2" filtered by provider include/exclude list, skipping libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: psm3 (707.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():557<info> "psm3" filtered by provider include/exclude list, skipping libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: mlx (1.4) libfabric:1780409:1744907435::core:core:ofi_reg_dl_prov():675<warn> dlopen(/projects/site/gred/smpg/software/oneAPI/2025.1/mpi/2021.15/opt/mpi/libfabric/lib/prov/libefa-fi.so): libefa.so.1: cannot open shared object file: No such file or directory libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: ofi_hook_noop (121.0) libfabric:1780409:1744907435::core:core:ofi_register_provider():530<info> registering provider: off_coll (121.0) libfabric:1780409:1744907435::core:core:fi_getinfo_():1449<info> Found provider with the highest priority mlx, must_use_util_prov = 0 [0] MPI startup(): max_ch4_vnis: 1, max_reg_eps 64, enable_sep 0, enable_shared_ctxs 0, do_av_insert 0 [0] MPI startup(): max number of MPI_Request per vci: 67108864 (pools: 1) libfabric:1780409:1744907435::core:core:fi_getinfo_():1449<info> Found provider with the highest priority mlx, must_use_util_prov = 0 [0] MPI startup(): libfabric provider: mlx libfabric:1780409:1744907435::core:core:fi_fabric_():1745<info> Opened fabric: mlx libfabric:1780409:1744907435::core:core:fi_fabric_():1756<info> Using mlx provider 1.21, path:/projects/site/gred/smpg/software/oneAPI/2025.1/mpi/2021.15/opt/mpi/libfabric/lib/prov/libmlx-fi.so [0] MPI startup(): addrnamelen: 1024 Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack: MPIR_Init_thread(196)........: MPID_Init(1719)..............: MPIDI_OFI_mpi_init_hook(1741): MPIDU_bc_table_create(340)...: Missing hostname or invalid host/port description in business card Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack: MPIR_Init_thread(196)........: MPID_Init(1719)..............: MPIDI_OFI_mpi_init_hook(1741): MPIDU_bc_table_create(340)...: Missing hostname or invalid host/port description in business card slurmstepd: error: *** STEP 16814.0 ON sc1nc124 CANCELLED AT 2025-04-17T09:30:36 *** srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: error: sc1nc124: tasks 0-11: Killed

ucx_info -v:

# Library version: 1.16.0 # Library path: /usr/lib64/libucs.so.0 # API headers version: 1.16.0 # Git branch '', revision 02432d3 # Configured with: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --enable-mt --disable-params-check --without-go --without-java --enable-cma --with-cuda --with-gdrcopy --with-verbs --with-knem --with-rdmacm --without-rocm --with-xpmem --without-fuse3 --without-ugni --with-cuda=/usr/local/cuda-12.2

ucx_info -d | grep Transport:

# Transport: self # Transport: tcp # Transport: tcp # Transport: tcp # Transport: sysv # Transport: posix # Transport: dc_mlx5 # Transport: rc_verbs # Transport: rc_mlx5 # Transport: ud_verbs # Transport: ud_mlx5 # Transport: cma # Transport: xpmem

mlx looks like it should be available from all of the outputs I see. This looks like a cluster configuration issue, but I don't know where to start troubleshooting. SLURM job scheduler is in use.

Re: Intel MPI mlx provider issue

TobiasK — Mon, 12 May 2025 13:34:33 GMT

@Antonio_D

Can you please provide the output of

fi_info

and also (without setting FI_PROVIDER)

I_MPI_DEBUG=30 I_MPI_HYDRA_DEBUG=1 mpirun IMB-MPI1 allreduce

Re: Intel MPI mlx provider issue

Antonio_D — Mon, 12 May 2025 15:36:35 GMT

See the attached files. We have recently found that the mlx provider works properly with mpirun, but not srun (when submitting to SLURM using an sbatch script).

Re: Intel MPI mlx provider issue

TobiasK — Mon, 12 May 2025 15:47:25 GMT

@Antonio_D

thanks can you please also attach the output of the failing srun command, e.g.:

I_MPI_DEBUG=30 I_MPI_HYDRA_DEBUG=1 srun IMB-MPI1 allreduce

Re: Intel MPI mlx provider issue

Antonio_D — Mon, 12 May 2025 16:06:21 GMT

See attached. When I run it with pmix using the following:

I_MPI_DEBUG=30 I_MPI_HYDRA_DEBUG=1 I_MPI_PMI_LIBRARY=/.../software/pmix/lib/libpmix.so srun --mpi=pmix IMB-MPI1 allreduce

I get the error, also attached.

Re: Intel MPI mlx provider issue

TobiasK — Fri, 16 May 2025 08:54:34 GMT

Can you please also provide the log with I_MPI_DEBUG=1000 ?

Re: Intel MPI mlx provider issue

Antonio_D — Fri, 16 May 2025 15:57:16 GMT

See attached for files with I_MPI_DEBUG=1000

Re: Intel MPI mlx provider issue

TobiasK — Fri, 16 May 2025 16:21:57 GMT

@Antonio_D
can you please try to set

I_MPI_HYDRA_IFACE=ib0

Also since it is working with mpirun, is slurm correctly setup, e.g. are you setting:

PropagateResourceLimitsExcept=MEMLOCK

https://slurm.schedmd.com/faq.html#memlock

Re: Intel MPI mlx provider issue

Antonio_D — Fri, 16 May 2025 16:20:37 GMT

Same result with I_MPI_HYDRA_IFACE=ib0

topic Re: Intel MPI mlx provider issue in Intel® MPI Library

Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue

Re: Intel MPI mlx provider issue