I was wondering if there is a possiblity to redirect and split stdout of each mpi rank to a seperate file, so that I can investigate the output of each mpi rank seperately.
Is your program written in C/C++ or Fortran. Both can replace stdout with a file (i.e file with rank number embedded).
Fortran has interoperablee function calls. You can call C from Fortran, and link in an object file generated by C/C++. This is relatively easy to do to get what you want.
I don't know if it is possible to do this directly with a command of the form: "mpirun mpi_binary.exe", but it should be easy enough to do if you add a "wrapper" script in the middle.
Most of my MPI jobs are launched with commands of the form: "mpirun wrapper.sh mpi_binary.exe".
In this case, mpirun launches one instance of "wrapper.sh" for each MPI rank. In the wrapper.sh script, I set up output file names based on the hostname and MPI rank number, then execute the "mpi_binary.exe" with rank-specific IO redirection.
A simple example might be (untested):
#!/bin/bash MYHOST=`hostname -s` MYRANK=$PMI_RANK # this variable may depend on your execution environment PERFOUT=perf.$MYHOST.$MYRANK STD_OUT=stdout.$MYHOST.$MYRANK STD_ERR=stderr.$MYHOST.$MYRANK # assume that the arguments to this script are the name of the executable # followed by any of its required options perf stat -o $PERFOUT $* 2> $STD_ERR 1> $STD_OUT