I have been trying to run VASP in parallel. But I am having running it in parallel on more than one computing nodes. It runs perfectly fine on one node but when I increase the number of nodes, the simulation crashes. I have gathered all the possible information from the execution and MPI calls with these options of mpirun: -v -check_mpi -genv I_MPI_DEBUG 5. The resulting information is in the attached files.
Please see if you can figure the exact problem and help me to sort out the problem. I will really appreciate any help.
As far as I see there's a message "RLIMIT_MEMLOCK too small" in the out.txt - this means that current system limits (mainly 'max locked memory') are insufficient.
You can check the system limits with 'ulimit -a'. You need to set 'max locked memory' to 'unlimited' (administrators permissions and cluster reboot may be required).
You can find the similar example in the Intel MPI Library Troubleshooting Guide.