Using srun and Intel MPI

[Kevin_McGrattan]

I am trying to run Intel MPI jobs using srun. I can run with mpirun, but I'd like to see if srun performs better. We have had some problems with hanging jobs. 

According to the Slurm User's Guide , I should add 

$ export I_MPI_PMI_LIBRARY=/path/to/slurm/lib/libpmi2.so
$ srun user_app.bin

to my sbatch script. But I cannot find libpmi2.so on my system, even though I have the latest slurm and Intel MPI Installed. Is this guidance still sound?

[TobiasK]

@Kevin_McGrattan 
please search for libpmi2 package in your OS.
With srun we are loosing some information on topology so we still prefer running with mpirun.

[Kevin_McGrattan]

mcgratta@spark-login ~ $ srun --mpi=list
MPI plugin types are...
        none
        cray_shasta
        pmi2

Even though libpmi2.so is not installed with our version of Slurm (23.11.6), I can run jobs with srun --mpi=pmi2.  But in cases where a job is split across nodes, the job will not start, claiming that 

MPI startup(): Pinning environment could not be initialized correctly. Intel MPI process pinning will not be used.
               Possible reason: Using Slurm's srun or other job submission commands from other job schedulers to launch your MPI job. In this case, job scheduler specified pinning will be used.

 So srun is working sometimes, but not always, and I'm wondering if this missing libpmi2.so has something to do with it.

[TobiasK]

@Kevin_McGrattan For pmi2 to function correctly, you definitely need libpmi2.