split output by mpi rank

[Timo_W_]

Hi all,

I was wondering if there is a possiblity to redirect and split stdout of each mpi rank to a seperate file, so that I can investigate the output of each mpi rank seperately.

best regards,

Timo

[jimdempseyatthecove]

Is your program written in C/C++ or Fortran. Both can replace stdout with a file (i.e file with rank number embedded).

https://stackoverflow.com/questions/18086193/redirect-stdout-stderr-to-file-under-unix-c-again

Jim Dempsey

[Timo_W_]

Dear Jim,

it is written in Fortran

best regards

Timo

[jimdempseyatthecove]

Fortran has interoperablee function calls. You can call C from Fortran, and link in an object file generated by C/C++. This is relatively easy to do to get what you want.

Jim Dempsey

[McCalpinJohn]

I don't know if it is possible to do this directly with a command of the form: "mpirun mpi_binary.exe", but it should be easy enough to do if you add a "wrapper" script in the middle.

Most of my MPI jobs are launched with commands of the form: "mpirun wrapper.sh mpi_binary.exe".

In this case, mpirun launches one instance of "wrapper.sh" for each MPI rank.  In the wrapper.sh script, I set up output file names based on the hostname and MPI rank number, then execute the "mpi_binary.exe" with rank-specific IO redirection.

A simple example might be (untested):

#!/bin/bash

MYHOST=`hostname -s`
MYRANK=$PMI_RANK           # this variable may depend on your execution environment

PERFOUT=perf.$MYHOST.$MYRANK
STD_OUT=stdout.$MYHOST.$MYRANK
STD_ERR=stderr.$MYHOST.$MYRANK

# assume that the arguments to this script are the name of the executable
# followed by any of its required options

perf stat -o $PERFOUT $* 2> $STD_ERR 1> $STD_OUT

[Timo_W_]

Dear John,

this might help me. I will investigate the example and try to adapt it to my case. Thanks!

best,

Timo