mpiexec error

[rigveds]

After compiling the code with ifort. while running job, i get following error. Please sugggest on wt can be the issue? how to vercome it?

i_1]: aborting job:-331x221.x
1: Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
1: MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0xc39a9a8) failed
1: MPI_Comm_rank(64).: Invalid communicator
2: [cli_2]: aborting job:
2: Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
2: MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0xc39a9a8) failed
2: MPI_Comm_rank(64).: Invalid communicator
3: [cli_3]: aborting job:
3: Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
3: MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0xc39a9a8) failed
3: MPI_Comm_rank(64).: Invalid communicator
0: [cli_0]: aborting job:
0: Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
0: MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0xc39a9a8) failed
0: MPI_Comm_rank(64).: Invalid communicator
10: [cli_10]: aborting job:
10: Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
10: MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0xc39a9a8) failed
10: MPI_Comm_rank(64).: Invalid communicator
11: [cli_11]: aborting job:
11: Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
11: MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0xc39a9a8) failed
11: MPI_Comm_rank(64).: Invalid communicator
rank 2 in job 1 compute-0-8_58700 caused collective abort of all ranks
exit status of rank 2: return code 13
rank 1 in job 1 compute-0-8_58700 caused collective abort of all ranks
exit status of rank 1: return code 13

machine is rehel 5.2 x86_64

[TimP]

You don't even mention which MPI version you have. Most of the better ones have helpful FAQ suggestions for debugging on their web sites. If you are using Intel MPI, the companion forum on HPC would be appropriate.

[Carlos_Antonio_Ribei]

Recently I was trying to compile and run my mpi code on a single machine (Ubuntu 12.04 - 64 bits core i7 2670 QM) I installed mpich2 version 1.2 using the following configuration: ./configure --prefix=/opt/mpich2 --enable-f77 --enable-fc --enable-cxx --with-device=ch3:sock --with-pm=mpd CC=icc CXX=icpc F77=ifort FC=ifort 2>&1 | tee configure.log The installation was ok, and I got mpd working well, I tested mpd with the examples and all is perfect. I compile my code using mpif90 and I got the code compiled with no errors. The flags I'm using to compile the code are: For the compiler: LN_FLAGS= -lm -larpack -lsparskit -lfftw3 -lrt -llapack -lblas For MPI linker: LN_FLAGS_MPI= $(LN_FLAGS) -I$(MPIHOME)/include -L$(MPIHOME) $(MPIHOME)/lib/libmpich.a -lfmpich -lopa -lmpe So the problem is when I try to run the code on my machine: First I invoke mpd as: mpd & and then run the code as: mpirun -np 4 ./code_mpi I tried a lot of variations as: mpiexec -np 4 ./code_mpi mpirun -n 2 ./code_mpi mpiexec -n 2 ./code_mpi And all results in the same error: Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator [cli_2]: aborting job: Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator [cli_1]: aborting job: Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator rank 2 in job 1 ubuntu_38132 caused collective abort of all ranks exit status of rank 2: killed by signal 9 Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator [cli_3]: aborting job: Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator [cli_0]: aborting job: Fatal error in MPI_Comm_rank: Invalid communicator, error stack: MPI_Comm_rank(106): MPI_Comm_rank(MPI_COMM_NULL, rank=0x14b46a0) failed MPI_Comm_rank(64).: Null communicator rank 1 in job 1 ubuntu_38132 caused collective abort of all ranks exit status of rank 1: return code 1 The code starts to run... but in 2 seconds give me this error! I spent almost 2 weeks trying to solve this problems because I really need to run this code in my personal computer to work at home. I appreciate all that can help me!

[TimP]

If you trot out your web search engine and loop up this mpich error message, you will see that a common (but far from only) cause is from getting different MPI implementations mixed up. If you installed OpenMPI in Ubuntu, a good move might be to remove it, and if you ever want it again build your own version to install in separate path, e.g. /opt/ompi. You really should check out the suggestions in the FAQs and web search references to narrow down your problems. If you find the level of help available for mpich insufficient, you might consider an MPI with more help resources.