Using srun and Intel MPI

Developer Software Tools Intel® MPI Library
,
1

I am trying to run Intel MPI jobs using srun. I can run with mpirun, but I’d like to see if srun performs better. We have had some problems with hanging jobs.

According to the Slurm User’s Guide , I should add

$ export I_MPI_PMI_LIBRARY=/path/to/slurm/lib/libpmi2.so
$ srun user_app.bin

to my sbatch script. But I cannot find libpmi2.so on my system, even though I have the latest slurm and Intel MPI Installed. Is this guidance still sound?

2

@Kevin_McGrattan  
please search for libpmi2 package in your OS.
With srun we are loosing some information on topology so we still prefer running with mpirun.

3 

mcgratta@spark-login ~ $ srun --mpi=list
MPI plugin types are...
        none
        cray_shasta
        pmi2

Even though libpmi2.so is not installed with our version of Slurm (23.11.6), I can run jobs with srun --mpi=pmi2.  But in cases where a job is split across nodes, the job will not start, claiming that

<span>MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.</span>
<span>               Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.</span>

So srun is working sometimes, but not always, and I’m wondering if this missing libpmi2.so has something to do with it.

4

@Kevin_McGrattan  For pmi2 to function correctly, you definitely need libpmi2.