error Pmpi_init error on linux

Mitul1 — Wed, 02 Oct 2024 15:42:47 GMT

While running job on HPC Rocky 8 with OpenHPC 2.x. Getting an error with Intel MPI

Error

Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:

MPIR_Init_thread(176)........:

MPID_Init(1538)..............:

MPIDI_OFI_mpi_init_hook(1511):

open_fabric(2568)............: OFI fi_getinfo() failed (ofi_init.c:2568:open_fabric:No data available)

Slurm JOB

#!/bin/bash

#SBATCH -p normal

#SBATCH --no-requeue

#SBATCH -J JOB_aimD

#SBATCH -o JOB_aiMD.%J

#SBATCH -n 40

ulimit -s unlimited

module purge

module list

module load intel mpi

Am I missing a library or package

Re: error Pmpi_init error on linux

taehunkim — Fri, 18 Oct 2024 06:43:49 GMT

Hi,

The error you're encountering suggests that the Intel MPI library is having trouble initializing the OFI (Open Fabrics Interfaces) provider, which is often used for high-performance networking in HPC environments. The specific error indicates that the OFI provider could not find any suitable network interfaces or fabric services.

Here are some steps to troubleshoot and resolve this issue:

Check OFI Providers: Ensure that the necessary OFI providers are installed on your system. You can check the available providers by running:

$ fi_info

This command should list the available fabric interfaces. If it returns "No data available," it means no suitable providers are found.

Install Required Packages: Make sure that the required OFI libraries and providers are installed. On Rocky Linux 8 with OpenHPC, you might need to install packages like libfabric and its providers. You can install them using:

$ sudo yum install libfabric libfabric-devel

Configure Intel MPI to Use a Specific Provider: Sometimes, specifying a particular provider can help. You can set the FI_PROVIDER environment variable to a specific provider that is available on your system. For example:

$ export FI_PROVIDER=sockets

You can add this line to your Slurm job script before the mpirun or srun command.

Check Network Configuration: Ensure that the network interfaces on your nodes are properly configured and accessible. The OFI provider might be looking for specific high-performance network interfaces (like InfiniBand or Omni-Path) that are not configured or available.

Intel MPI Configuration: Intel MPI can be configured to use different communication fabrics. You can try setting the I_MPI_FABRICS environment variable to use a different fabric. For example:

$export I_MPI_FABRICS=shm:ofi or export I_MPI_FABRICS=shm:tcp

Add this line to your Slurm job script before the mpirun or srun command.

You can get hint at here ( https://www.intel.com/content/www/us/en/docs/mpi-library/developer-reference-linux/2021-13/ofi-capable-network-fabrics-control.html )

Thanks.

topic error Pmpi_init error on linux in Intel® MPI Library

error Pmpi_init error on linux

Re: error Pmpi_init error on linux